Live data from Hacker News

Ask HN: Recommended resources to learn the Linux kernel and OS theory?

news.ycombinator.com

71–80 of 84 posts

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#71

The OSDev wiki can probably teach you anything you need to know about "OS theory" (and practice, mostly on x86): https://wiki.osdev.org/Expanded_Main_Page Depending on your style of programming, I'd recommend maybe not working on Linux if you are trying to learn kernel development and OS internals. Have you looked at Haiku ( https://www.haiku-os.org/ )? We have a very well organized and commented modular-monolitic ke…

> modular-monolitic kernel

This sounds like a bit of an oxymoron. What is a modular-monolithic kernel?

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#72
In school we used Tannenbaum's MINIX book (Operating Systems Design and Implementation). It was generally a pretty good introduction to "OS Theory", directly references the source code in the back. Useful for a desk book if you're constantly elbows deep in writing syscalls.

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#73
For learning the basics about how an operating system works, I recommend Remzi and Andrea Arpaci-Dusseau's "Three Easy Pieces", which is freely available at http://pages.cs.wisc.edu/~remzi/OSTEP/. Both Remzi and Andrea Arpaci-Dusseau are great writers and are prolific researchers in the areas of operating systems and file systems, and their work is top-notch.

For learning the basics about how the kernels of Unix-like systems are designed, I recommend the following books:

- The Design of the UNIX Operating System (1986) by Maurice Bach. You could also compare this book to the source code of Version 7 Unix, which is freely available under a BSD-style license. Many of the basic design principles of Unix-like operating systems have not changed dramatically over the years, and so studying the design and implementation of older, smaller implementations of Unix-like operating systems can aid with figuring out the code base of Linux, which is huge.

- UNIX Internals: The New Frontier (1996) by Uresh Vahalia. This discusses some more modern features that weren't part of AT&T Unix.

- The Design and Implementation of the FreeBSD Operating System (2014) by Marshall Kirk McKusick et al. I've met "Kirk" in person a few times at USENIX FAST conferences; he's not only a great writer, but he is also very friendly and approachable. This is a well-written text describing FreeBSD 11, including modern features such as ZFS.

There are some good Linux kernel books (I'm partial to "Linux Kernel Development" by Robert Love, which helped me out nine years ago when I was writing a device-mapper device), but what I've found during my experience working with the Linux kernel is that the Linux kernel changes much faster than the books describing it get updated. Nevertheless, the best thing to do is to get a solid understanding of how Unix works, then get a basic understanding of the Linux kernel, then start diving into the source code, which is ultimately the definitive documentation of the Linux kernel.

When I was working on that device-mapper device, I found that perusing the Linux kernel mailing list was excellent for learning about how relevant portions of the kernel changed over time.

Good luck and have fun! I haven't done much kernel-level work lately, but I miss working at that level!

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#74

The OSDev wiki can probably teach you anything you need to know about "OS theory" (and practice, mostly on x86): https://wiki.osdev.org/Expanded_Main_Page Depending on your style of programming, I'd recommend maybe not working on Linux if you are trying to learn kernel development and OS internals. Have you looked at Haiku ( https://www.haiku-os.org/ )? We have a very well organized and commented modular-monolitic ke…

> modular-monolitic kernel This sounds like a bit of an oxymoron. What is a modular-monolithic kernel?

The Linux kernel, as well as the FreeBSD kernel, supports the loading and unloading of kernel modules, which are compiled binaries that can be linked to the kernel while it is running. These modules are still in the address space of the kernel; the term "monolithic" refers to the fact that all portions of the kernel, even kernel modules, are running in the same address space. This is different from a microkernel like Mach where various operating system subsystems such as networking and the file system are running as separate user-level processes (thus having their own address spaces).

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#75

One important part about learning how the Linux kernel works is understanding the details of the system's interface the kernel provides. In my opinion, there is no better book out there than the "Linux Programming Interface" by Michael Kerrisk: http://man7.org/tlpi/ It provides extremely detailed information about everything going on in Linux, as well as example programs and exercises to help you further your knowled…

That’s a fantastic book.

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#77
I've semi-seriously been considering starting my own OS project to circumvent some of the IO latency issues with VR and AR applications.

The big metric is motion-to-photon latency, which is a measure (in milliseconds) of how quickly a user's head motion is reflected on the display. Anything greater than ~10ms is considered insufficient, as you've missed a whole frame of rendering for VR headsets that run at 90hz refresh rate.

Current XR applications need to wait for the USB or Bluetooth stack to return the head pose before modifying the view, then wait for the graphics stack to reflect that change. There are many ring transitions along the way. This leads to an absolute minimum motion-to-photon latency that can't be broken just because of OS overhead. This eats into your frame budget, as every ms of context-switching out of the IO drivers leaves you fewer ms to render a frame in time to get it back into the kernel for context-switching back to the graphics driver.

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#78

"Operating Systems: Three Easy Pieces" is a more modern book to OS concepts, explained with real source code as opposed to algorithms, and is regularly updated. http://pages.cs.wisc.edu/~remzi/OSTEP/ (Online version above, you can also order a print copy).

ostep.org is an easy-to-remember shortcut

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#79

I've been making my way through this course https://classroom.udacity.com/courses/ud923 It's about 9 years old I think so might be too out of date for you, but I've been really enjoying it. I think they have an advanced version of the course as well if you're already familiar with the basics

That appears to be the Intro to Operating Systems course from the Georgia Tech OMSCS program.

Udacity also hosts the GT Advanced Operating Systems course: https://www.udacity.com/course/advanced-operating-systems--u.... AOS is not a "how to" course, but rather a tour of the challenges operating systems may face (distribution, fault tolerance, scale) and of academic papers attempting to solve them (mostly from the 90s and 00s). As such, the material might not be a good fit for the original post, but this commenter found it fascinating.

The reading list is here: https://www.udacity.com/wiki/ud156-readings

Re: Ask HN: Recommended resources to learn the Linux kernel and OS theory?

#80
post #11

Yes, there is just so much, and it is not thoroughly documented. "BPF maps" in particular are a) a special feature of a special feature that's Linux-specific, not anything that generalizes across OSes, and b) a feature for userspace anyway, not for kernel internals. But even longstanding kernel-internal features aren't well documented (e.g., the other day I was trying to figure out what struct file's f_version does,…

Perhaps I should've said more. I do often end up googling basic OS stuff, especially related to memory (software interrupts still confuse me though, I understand the concepts, but never the implementation), but even more so have been my ignorance of hardware level concepts. For example trying to patch the code for a network card the other week, I learned what watchdog timer is, I've never heard of that prior.

Working through an academic OS class where you implement interrupt handling is super helpful for that - hardware interrupts and software interrupts are similar but not quite the same and actually writing the interrupt handlers makes some things clear.

I can endorse MIT's 6.828 (I've taken it and taught a version if it), which (at least before this semester, they seem to be redoing things) has a lab about setting up interrupt handling: https://pdos.csail.mit.edu/6.828/2018/labs/lab3/ It's entirely possible to complete the labs on your own, and doing labs 1-3 is worthwhile, but it's also a significant time commitment.

Watchdog timers fall in the category of "there is so much" - they're not crucial to how network drivers work, but drivers in a production OS have to care about every feature that might interact with them. :(

Post reply on HN