CS 450: Operating Systems

Staff

Instructor: Kyle C. Hale
  Office Hours: Wed 3:05-5:05 PM (SB 229C)
  E-mail: khale [at] cs [dot] iit [dot] edu

TAs:

Amal Rizvi
  Office Hours: WF 10-11:30AM (SB 115)
  E-mail: arizvi1 [at] hawk [dot] iit [dot] edu

Boyang Li
  Office Hours: T 3:00-4:00PM, Th 1:30-3:00PM (SB 007B)
  E-mail: bli70 [at] hawk [dot] iit [dot] edu

Course Info

Course number: CS 450

Semester: Spring 2019

Lecture Time: Mon/Wed 1:50PM - 3:05PM

Lecture Location: Pritzker 111

Overview

Operating systems run the world. Without an OS, you couldn't have hundreds of programs running at once, you'd have to have deep understanding of hardware, and you'd have to rewrite your code for each new machine on the market. You wouldn't have web servers, desktop machines, laptops, phones, or smart watches; no Siri, no Alexa, no TensorFlow or CUDA programs.

In this course, you will dive straight into the deepest and often most confounding component of a computer system's software stack. You will become familiar with both the high-level concepts underpinning modern operating systems and with low-level mechanisms, in addition to new and emerging techniques and programming models for computer systems. I am a strong believer in learning by doing, so the majority of your time for this course will be spent working on programming projects, both very low-level work and high-level (user-space) work. This course will be challenging! It will bring together a lot of concepts you've learned in your previous courses, from data structures and algorithms to assembly language, computer architecture, virtual memory, and advanced C programming.

For this course, I assume that you have basic familiarity with computer organization (as introduced in CS 350), data structures, C, and systems programming (as covered in CS 351). You will for sure need to be able to program in C and should be able to navigate around a UNIX environment.

Communication

We will be primarily using Piazza as a course communication mechanism. If you have an issue or question that is not strictly private (especially one that would benefit everyone were it answered), please use Piazza as your first resource. The instructor, the TA, and your fellow classmates will be there to help. Note that you can also post anonymously if you so choose.

Homeworks

While there will be no homework to turn it, it is a very good idea to do some of the homeworks associated with the readings as I post them. These are usually short coding tasks that use a simulator for the OS component in question. These homeworks will help solidify knowledge needed for the exams.

Readings

In addition to the projects (and homeworks if you're studious), you'll be needing to do the readings I post (almost) every lecture. These will primarily come from the OSTEP book, but I will also post other required and optional readings from seminal research papers, articles, and books.

We will, when time allows, also be spending some time reading code! This will likely include code snippets from real OSes including the Linux kernel.


The following is a rough schedule of the class. Note that it is subject to change.

Lecture Schedule

Week Date Item Topic Notes Reading
1 Mon 1/14 Lec 1 Intro and Administrivia slides; p1a out Required Reading: OSTEP Preface, 1, 2
1 Wed 1/16 Lec 2 Resource Virtualization and Processes slides Required Reading: OSTEP 3, 4, 5, 6, System Calls
2 Mon 1/21 holiday MLK Day (No class) p1a in, p1b out
2 Wed 1/23 Lec 3 Scheduling slides Required Reading: OSTEP 7, 8, Summary, Linux processes, Interrupts (gentle)
Optional Reading: A decade of wasted cores, OSTEP: Multi-processor Scheduling, Linux CFS
3 Mon 1/28 Lec 4 Virtual Memory slides
follow-up video
Required Reading: OSTEP 13, 15, 16, Anatomy of a program in memory
Optional Reading: Kernel memory management, Closures, stacks, and the heap
3 Wed 1/30 -- Arctic Interrupt: University closed Required Reading: OSTEP 18, 19
4 Mon 2/4 Lec 5 Intro to Paging slides
p1b in
Required Reading: OSTEP 20, Getting Physical with Memory
Optional Reading: Memory Translation and Segmentation
4 Wed 2/6 Lec 6 Multi-level Paging and the TLB Guest lecturer; slides Required Reading: OSTEP 21, 22, Summary
Optional Reading: OSTEP 23, The Page Cache
5 Mon 2/12 Lec 7 Virtual Memory and Paging Tricks slides Required Reading: OSTEP 25, 26
5 Wed 2/14 Lec 8 Concurrency I: Threads slides Required Reading: OSTEP 28, 29
6 Mon 2/18 Lec 9 Concurrency II: Mutual Exclusion (locks) slides Required Reading: OSTEP 30
6 Wed 2/20 Lec 10 Concurrency III: Signaling and Condition Variables slides Required Reading: OSTEP 31, 33, Signals, Linux kernel signals
7 Mon 2/25 Lec 11 Concurrency IV: Semaphores, Signals, Event-driven architecture slides Required Reading: OSTEP 32, Sys V. Shared Memory, Sys V. Message Queues
7 Wed 2/27 Lec 12 Concurrency V: Concurrency Bugs and IPC slides Required Reading: OSTEP d, 36
8 Mon 3/4 Lec 13 I/O I: Devices and Drivers Required Reading: Overview of PCIe, Chipsets and the Memory Map
Optional Reading: How PCI Express Works
8 Wed 3/6 Lec 14 I/O II: PCI and dynamic MMIO Required Reading: OSTEP 37
Optional Reading: What your computer does while you wait
9 Mon 3/11 Lec 15 I/O III: Disks slides
Interested in research?
Required Reading: OSTEP 38
Optional Reading: Original RAID paper
9 Wed 3/13 Midterm (in class) (See Exams)
10 Mon 3/18 holiday Spring Break (No classes)
10 Wed 3/20 holiday Spring Break (No classes)
11 Mon 3/25 Lec 16 I/O IV: RAID slides Required Reading: OSTEP 38
11 Wed 3/27 Lec 17 File Systems I: File Abstractions slides Required Reading: OSTEP 39
12 Mon 4/1 Lec 18 File Systems II: Implementation slides Required Reading: OSTEP 40
12 Wed 4/3 Lec 19 File Systems III: FFS slides Required Reading: OSTEP 41
13 Mon 4/8 Lec 20 File Systems IV: Journaling and LFS slides Required Reading: OSTEP 42, 43
13 Wed 4/10 Lec 21 SSD, Flash, and NVM slides Required Reading: OSTEP 44, NVDIMMs
14 Mon 4/15 Lec 22 Distributed Systems I slides Required Reading: OSTEP 48
14 Wed 4/17 Lec 23 Distributed Systems II: RPC and NFS slides Required Reading: OSTEP 49, 50
15 Mon 4/22 Lec 24 Programming Models I: MapReduce slides Required Reading:MapReduce
15 Wed 4/24 Lec 25 Case Study: GFS slides Required Reading:Google FS
16 Mon 4/29 Lec 26 Case Study: Search Engines
16 Wed 5/1 Lec 27 Future Directions: Unikernels, Containers, Serverless Computing
-- Mon 5/6, 2-4PM Final Exam (same room) (See Exams)

Exams

Week Date Item Length Covers Notes
9 Wed 3/13 Midterm 50 min Lec 1-15
17 Monday 5/6, 2-4pm (same room) Final Exam 75 min Apx 75% Lec 16 on, 25% midterm material

Projects

You'll notice that (almost) every project listed below for this class is split into two parts. In many cases these are really entirely separate projects, but are logically grouped together to stress some particular concept. The (a) parts are mostly concentrated on userspace components (part of the broader operating system) and the (b) parts are kernel components that will involve hacking on xv6. 3a and 4a are subject to change based on an in-class vote. Unless otherwise noted, grades for these projects are going to be based on whether or not your code works. Partial solutions will not be given credit! With that in mind, you should always start early.

Project Topic Due Date Handout Notes (TBP = To Be Posted)
1a UNIX warmup Monday, 1/21/2019 11:59PM CST p1a description Posted 1/14/2019 5:00PM
1b xv6 warmup Monday, 1/28/2019 Monday, 2/4/2019 11:59PM CST p1b description Posted 1/21/2019 9:00PM
2a out of the depths of x86 Thursday, 2/14/2019 11:59PM CST p2a description Posted 2/4/2019 11:59PM
2b scheduling in xv6 Thursday, 2/28/2019 Friday, 3/1/2019 11:59PM CST p2b description Posted 2/14/2019 11:00PM
3a parallel zip Friday, 3/15/2019 11:59PM CST p3a description Posted 3/1/2019 7:00PM
3b null pointers and shared memory Friday, 3/29/2019 Monday, 4/1/2019 11:59PM CST p3b description Posted 3/12/2019
4a implementing kernel threads Monday, 4/15/2019 Wednesday, 4/17/2019 11:59PM CST p4a description Posted 4/1/2019
4b MapReduce Wednesday, 5/1/2019, 11:59PM CST p4b description Posted 4/17/2019

Books

The following book is the only required textbook for this course. It is not only an excellent textbook, but is also freely availble online:

Operating Systems: Three Easy Pieces by Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, 2015.

The below books are also great and often-used texts on Operating Systems. If you find a copy of one for a good price, snatch it. Keep in mind, however, that the best way to earn operating systems is by building!.

This is probably my favorite OS textbook after OSTEP. Full disclosure, I took undergrad OS with Dahlin many years ago.

Operating Systems: Principles and Practice (2nd Edition), by Thomas Anderson and Michael Dahlin, Recursive Books 2014.
Operating System Concepts (9th Edition), by Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne, 2012 Wiley.
Modern Operating Systems (4th Edition), by Andrew S. Tanenbaum and Herbert Bos, Pearson 2014.

For general systems design, a good one to have on the shelf:

Principles of Computer System Design: An Introduction (1st Edition), by Jerome H. Saltzer and M. Frans Kaashoek, Morgan Kaufmann 2009.

The following book, while outdated (uses a 2.6.10 kernel as a reference), still provides one of the best references for practical kernel hacking:

Understanding the Linux Kernel (3rd Edition), by Daniel P. Bovet and Marco Cesati, O'Reilly Media 2005.

Good book on Linux kernel programming:

Linux Kernel Development (3rd Edition), by Robert Love, Addison-Wesley Professional, 2010.

If you're interested in the macOS kernel, this one is not out yet, but will be soon:

OSX Internals, Volume II, 2nd Edition, by Jonathan Levin.

Similarly for Windows:

Windows Internals, Part 1: System Architecture... (7th Edition), by Pavel Yosifovich, Mark E. Russinovich, Alex Ionescu, and David A. Solomon, Microsoft Press 2017.

Development Environment

We will be using the Fusion cluster for development and testing. This means you will be doing most of your development remotely. You should all have the ability to log in to the cluster when classes start. You might also want to get used to using tools like tmux or sshfs. All the tools you will need are on Fusion.

Other Useful Links and Resources

This is a list of other resources that you might find useful for this class. Feel free to peruse them at your own convenience.

Links

Interesting, important, historical, and new OS Kernels

  • The Linux Kernel, arguably the most important kernel of our time
  • FreeBSD
  • Nautilus, a research OS written by your instructor for parallel runtime systems
  • Plan 9, an important (but commercially unsuccessful) UNIX successor by Bell Labs
  • Harvey OS, a new successor to Plan 9
  • Drawbridge, research prototype from MSR (components of which are now in the NT kernel).
  • MINIX 3, a historically important microkernel (originally---and still---used for teaching). Intel chips actually run this internally!
  • Several OSes based on the L4 kernel, an important early microkernel
  • Android OS, Google's mobile OS (based on Linux kernel, but probably soon Fuchsia (below))
  • Fuchsia OS, a new microkernel-based OS from Google
  • Barrelfish OS, an important research OS from ETH Zurich (the OS is a distributed system)
  • Redox, an OS written in Rust (UNIX-like)
  • Kitten Lightweight Kernel, an OS kernel designed for supercomputing
  • Scout, a network-oriented research OS
  • MirageOS, a Unikernel for OCaml applications
  • OSv, a popular new Unikernel
  • Pintos, a teaching OS based on the Nachos research OS.
  • JOS, another teaching OS
  • OS/2, an important mainframe OS from IBM
  • COS, an OS for some of the first supercomputers
  • CP/M, an OS for some of the first microcomputers. Here is the code.
  • code for the OS of Xerox Alto, the first computer with a GUI
  • source code for MS-DOS 1.25 and 2.0
  • source code for the XNU kernel, part of Darwin, which powers macOS
  • source code for ReactOS, a WindowsNT-compatible kernel
  • System software for the Apollo Guidance Computer
  • Code for the Apple II's DOS
  • Code for the Atari 7800's OS