Ask HN: How to learn proper systems programming?

There are two books that taught me how systems work.

- One system in isolation - Operating Systems: Three Easy Pieces. Covers persistence, virtualisation and concurrency. This book is available for free at https://pages.cs.wisc.edu/~remzi/OSTEP/

- Multiple systems, and how data flows through them - Designing Data Intensive Applications. Covers the low level details of how databases persist data to disk and how multiple nodes coordinate with each other. If you’ve heard of the “CAP theorem”, this is the source to learn it from. Worth every penny.

More on why these two books are worth reading at https://teachyourselfcs.com

First off, don't worry. You already have all the necessary tools you need.

Although the language you use to learn system & network programming doesn't matter much, it is better if you use C or C++ to practise and learn. This is because the kernel itself is written in C and exposes system calls that can be used directly from a C/C++ program. That said, "The Linux Programming Interface"(I am personally reading it) is a really good book. It talks a lot about how one should go about using system calls to get things done by the kernel. Make sure to read a little every day and try out the examples by writing C/C++ programs.

I recently realized that TLPI doesn't talk much about why are things the way they are(a very good example would be virtual memory and related stuff). You should refer some theoretical book for this. I suggest you go with "Operating systems" by Deitel & Choffnes.

Read man pages and practise using the libc/kernel APIs. For example, if you want to know about flushing, read 'man 3 fflush'. This might be needed when you want to flush all the input/output data that has been buffered by the C library before you can get fresh input from stdin. For example, if prompts are buffered, you definitely don't want to "scanf" before you have flushed the buffers. If you want to learn network programming, read chapters related to socket and refer 'man 2 socket'.

You will eventually get to a point where you will be able to connect all the dots(APIs) and be able to figure out what exactly you will need to get some problem solved.

Finally, don't learn for a future job. Learn for yourself. This will help you in the long run.

There are a few good YouTube channels of strong system programmers doing live coding. Watching videos takes time, but you can pick up lots of techniques, big and small, by watching people work.

Since you're looking at rust, https://www.youtube.com/user/gamozolabs/videos could be a good fit.

Plan9 is known for having very accessible code.

What worked for me is trying to develop my own kernel. This might be too low-level for you, but it helps immensely to dispel the magic around how code gets executed by the CPU and the OS. You'll learn how the OS achieves protection from user programs, what are interrupts and how does the OS handle them, what are processes and threads and how does the OS schedule their execution, how virtual memory works (segmentation/paging), how the program is laid out in memory (code, data, heap, stack), how runtimes like the C runtime manages memory and links your code to OS syscalls, how dynamic linking works, what is an ABI, what is a calling convention, how does the compiler, the linker, and the OS know what each should do? How memory mapped file access works, etc. I can keep going, but you get the picture.

From there, you'll know where to go next based on what you've learned so far.

My recommended reading list is:

[1] Operating Systems: Three Easy Pieces https://pages.cs.wisc.edu/~remzi/OSTEP

[2] Intel Software Developer Manuals (especially volume 1 and 3A) https://software.intel.com/content/www/us/en/develop/article...

[3] OSDev wiki https://wiki.osdev.org