Ask HN: How to learn proper systems programming?

Most POSIX APIs are crap, but most significant systems projects don't code to them directly. Almost all large projects have their own internal (or third party) libraries that either wrap or reimplement the underlying POSIX APIs.

That said, at some point knowing the POSIX APIs does become necessary, but that's usually relevant a bit later when you start becoming interested in modifying the toolkit of large projects.

If you're wanting to start a project on your own rather than jump into an existing project, in a nutshell, if you don't have hard real-time requirements, if you're using C, use glib, if you're using C++, use Qt. There are other libs that are useful for real-time usage (basically things that don't ever have hidden memory allocation), but I'm not familiar enough with that space to recommend one.

Any channels you can recommend that have live system coding in C?

Yep. Write a grep and find (IO-loop, filesystem), a shell (child processes, signals), a simple nmap-replacement (network, DNS), a forking and a multithreaded webserver (more network, more children, synchronisation), a clock with seconds display and something that synthesizes and plays music on keypress (timing, waiting).

Everything just in the simplest sense, no fancy features needed: the grep just needs a string parameter to search for and some files, the shell doesn't need completion, scripting or variables, just execution and background jobs. For the webserver, just serve some static files from the URL, ignore security, concentrate on getting lots of clients served at the same time however. Bonus points if you make the main process/thread gather meaningful statistics/logs and not screw up concurrency. Make sure to learn the right synchronisation primitives and use them properly. For the clock, make sure to explore sleep-based, loop-based and signal-based approaches and compare them. Get all three second-ticks to be in sync with your pocket watch and not skip/delay/hang. The music exercise is similar, just event-based (you get to handle key input and buffer low or tick events), either with something like select/epoll or multiple threads. The music itself is not interesting, a simple sine or rectangle signal suffices. But of course reaction time should be low and sound should be glitch-free.

What is missing from the above but important are the security aspects of systems programming, most of which are either problems with certain languages (learn how to avoid, recognize and exploit a simple buffer overflow, format string exploit), security aspects around file creation (especially temporary files, but also general symlink attacks), SUID-bits, permissions/ACLs/MAC and generally privilege separation. Those aren't easy to put into learning-by-doing exercises, because you would need an attacker to slap you over the head when you make a mistake there ;)

As a programming language, for Linux/Unix I would strongly recommend using C, not C++, not anything else. Use libc or plain syscalls, nothing fancier. When you have mastered the above in C, you know how to appreciate other, better languages, but also know where those may be lacking. If you just do the above in Python, you learned nothing about systems programming and everything about Python lib idiosyncracies

How is that not just good old “application programming”?

My own definition of “system programming” is exactly “developing on top of the system” vs “developing in the comfort of a helper library” (and its limits). I consider myself a decent application programmer, but I'm not a system programmer (at least not yet ;).

The OP was talking about Postgres, if you don't know how the pitfalls of write(2), or don't know how to use mmap(2), you're going to have trouble making a database on your own.

Come up with a project and write code. For example you can just implement the web server you were mentioning. That is a good start. Build it and see how to scale it to handle millions of requests. Just creating the code to test it will be a good exercise. You can dig into a lot writing a web server. After that you can pick up another project in the kernel.