Ask HN: What should a systems/low-level software engineer know?

Are the jobs still there or are they disappearing?

I am bored of the churn associated with stuff higher up the stack. I would prefer to master a craft rather than chase new fads every few years.

Write a kernel extension. The Mac OS documentation is very comprehensive and helpful. It's a different type of programming to most people's day-to-day stuff. And it's very rewarding to see it load up into the OS (or crash the whole system as the case may be).

You should learn C. Ideally, you could learn C++ as well, but I think it's best to learn C first. And then treat C++ as a completely separate language. Rust is a good thing too, but it's a little too soon for it to be broadly useful.

Then you should learn Unix. From an understanding point of view, I think it's probably better to learn something like FreeBSD, NetBSD, Xv6 ... Linux is very pragmatic, and very general, and so it doesn't have the purity that smaller, more focused or curated systems have. Once you have a handle on Unix, look at other OSes: Plan9, Minix, FreeRTOS, L4, etc.

Then networking: I suggest starting with IP, TCP, ARP; then step down to the physical layer: Ethernet, Token Ring, 802.11, hubs, and switches; then static routing, RIP, OSPF, and BGP; maybe look at mesh routing. Then some application layer stuff: DNS, NTP, SSH, NFS, LDAP, HTTP, etc. Reading the RFCs is really valuable, and they're remarkably accessible up until say 2500 or 3000 or so.

Security: symmetric and asymmetric crypto, Kerberos, SSL, SSH, OAuth, etc. Read up on pen testing, social engineering, defensive programmings, fuzzing, etc.

Databases: both relational and otherwise. SQL. Wrap your head around how a filesystem and a database are the same and how they're different.

Messaging: some sub-set of protocol buffers, Cap'n'Proto, Avro, Thrift, XDR; brokers vs. p2p; pub-sub vs. directed. There are hundreds of system you can look at here: pick a few that look different.

Learn about complexity analysis, distributed consensus, locking, concurrency and threads.

so far as tools go, you need to understand a debugger (how to use it, and how it works), packet capture and analysis (Wireshark is good), profiling and performance analysis.

That's probably a decent coverage for the software side. The exact focus will differ depending on embedded/real-time vs enterprise, etc.

From the hardware side, I think it's worth starting with an 80's or earlier, 8 or 16-bit system, and learning how it works to the digital logic level. What a simple microprocessor actually does: fetching, decoding, execution units, etc. A Z80 or 6502 or similar system is a pretty simple circuit, and it's worth really grokking how it works.

From there, you can move forward to more complex CPUs, newer memory architectures, newer buses, etc. But it's much harder to dive straight into a modern x86 or ARM CPU and try to understand how it works.

It's a this point that reading Drepper's memory article, and the "everything you should know about latency" article(s), etc, really start to be useful, because you've got a solid grounding in what's underneath them.

You don't need to do this all at once, or before you start working more on backend or systems level code: I'd guess it took me close to 10 years to feel like I had a decent grasp on most of it.

Assuming that you want to move into embedded.

Focus on C and follow up with an understanding of the underlying assembler code. Haven't played with Rust for about 1 year, so I don't how how mature it is these days.

Also have an idea of how to debug the code. Debugging on embedded is very different to debugging a web/js app.

Low-level software often moves quite slowly and backwards compatibility is important. Embedded systems have to be maintained for decades. So knowing how do handle legacy code, proper dependency management, release management, and requirement management is quite important. Similarly the cost of bugs is usually quite high, because updates are hard. So writing testable code, testing and other QA activities are more important.