Ask HN: What should a systems/low-level software engineer know?

Does knowledge of the STM32 family apply to other ARM chips?

A systems programmer will specialize in subdomains much like any other programmer, but there are some characteristic skills and knowledge that are common across most of them. Systems programmer coding style is driven by robustness, correctness, and performance to a much greater degree than higher up the stack. Most systems programming jobs these days is C/C++ on Linux and will be for the foreseeable future.

- Know how your hardware actually works, especially CPU, storage, and network. This provides the "first principles" whence all of the below are derived and will allow you to reason about and predict system behavior without writing a single line of code.

- Understand the design of the operating system well enough to reimplement and bypass key parts of it as needed. The requirements that would cause you to to build, for example, a custom storage engine also means you aren't mmap()-ing files, instead doing direct I/O with io_submit() on a file or raw block device into a cache and I/O scheduler you designed and wrote. Study the internals of existing systems code for examples of how things like this are done, it is esoteric but not difficult to learn.

- Locality-driven software design. Locality maximization, both spatial and temporal, is the source of most performance in modern software. In systems programming, this means you are always aware of what is currently in your hardware caches and efficiently using that resource. As a corollary, compactness is consistently a key objective of your data structures to a much greater extent than people think about it higher up the stack. One way you can identify code written by systems programmer is the use of data structures packed into bitfields.

- Understand the difference between interrupt-driven and schedule-driven computing models, the appropriate use cases for both, and how to safely design code that necessarily mixes both e.g. multithreading and coroutines. This is central to I/O handling: network is interrupt-driven and disk is schedule-driven. Being explicit about where the boundaries are between these modes in your software designs greatly simplifies reasoning about concurrency. Most common concurrency primitives make assumptions about which model you are using.

- All systems are distributed systems. Part of the systems programmers job is creating the illusion that this is not the case to the higher levels in the stack, but a systems programmer unavoidably lives in this world even within a single server. Knowing "latencies every programmer should know" is just a starting point, it is also helpful to understand how hardware topology interacts with the routing/messaging patterns to change latency -- tail latencies are more important than median latencies.

The above is relatively abstract and conceptual but generalizes to all systems programming. Domains of systems programming have deep and unique knowledge that is specific to the specialty e.g. network protocol stacks, database engines, high performance graphics, etc. Because the physics of hardware never changes except in the details, the abstractions are relatively thin, and the tool chains are necessarily conservative, systems programming skills age very well. The emergence of modern C++ (e.g. C++17) has made systems programming quite enjoyable.

Also, the best way to learn idiomatic systems programming is to read many examples of high-quality systems code. You will see many excellent techniques in the code that you've never seen before that are not documented in book/paper form. I've been doing systems work for two decades and I still run across interesting new idioms and techniques.

I'd start with everything by Hennessy and Patterson. That should cover most of how CPUs and memory systems work.

I grew up with Tannenbaum for networking, though most people went toward Comer and Stevens. Maybe there's something even more current.

I honestly don't know anything as good for storage, which is funny since it's my own specialty. I can try to cobble together a reading list if you'd like.

Something on operating systems. Vahalia for an overview, Bach for System V, gang of four for BSD. Yes, study something besides Linux ... but do study Linux as well. Not sure which books are good for that.

Something on compilers. Is the dragon book still the go-to reference here?

Even if you don't work in those specific areas, that should give you the grounding to study more. I'd also throw in something on databases, but don't know a specific book/author. Distributed systems much like storage: definitely learn, can't think of a good single source, can create a list if you want (I think I did one for a colleague not too long ago).

Good luck!

For High Throughput compute, and cache correctness. Here are the primers I can give. None of these are required. Strong langauge, cs, interpersonal skills, team inter-interoperability, and commitment to quality (via unit and integration testing, and skill working with liners) can get you a lot further.

The resources I'm linking are supplementary to the above, and you'll likely encounter them in the wild. But they'll help you build a base of knowledge, and give you terminology to search for, and work with.

- What Every Programmer should know about memory: https://people.freebsd.org/~lstewart/articles/cpumemory.pdf

- Cache Obviousness: https://www.youtube.com/watch?v=bY8f4DSkQ6M (the suggestions, and terminology are important)

- Parallelism: (This is a good primer, there are a lot of complementary posts linked on the site) https://preshing.com/20120612/an-introduction-to-lock-free-p... https://preshing.com/20120913/acquire-and-release-semantics/

- If you plan on working with linux these is an excellent reference: http://man7.org/linux/man-pages/dir_section_2.html remember there is no magic in Linux, everything eventually has to go through a system call. So if you learn the systemcalls, you can can learn how things work :)

- Fog's optimization resources are awesome: https://www.agner.org/optimize/

- MIT courseware: https://www.youtube.com/watch?v=ytpJdnlu9ug&list=PLUl4u3cNGP... https://www.youtube.com/watch?v=HtSuA80QTyo&list=PLUl4u3cNGP...

- This cheat sheet is worth committing to memory: http://www.bigocheatsheet.com/ the reference links are also great for building up knowledge

- I highly recommend CMU DB open course ware: (intro) https://www.youtube.com/watch?v=vyVGm_2iFwU&list=PLSE8ODhjZX... (advanced) https://www.youtube.com/watch?v=poEfLYH9W2M&list=PLSE8ODhjZX...

This should give you a good primer on Concurrency, and DB's. For networking likely a basic TLA+ certification class will be 99% review, but for the things it isn't will offer great insight.

Emphasis on 8. If you are writing code that directly controls hardware, then your software is the abstraction.

Hardware is physical and can be quirky. Signals take real clock time to settle down and your CPU is probably much faster than the transition time. Hardware can and does have bugs just like software. A bad ground pin can make your hardware do “impossible” things.

You need to have much more awareness of the context your code runs in the further down the stack you go.