The Deadlock Empire: An Interactive Guide to Locks

The lowest level is understanding things on the hardware memory model level--knowing how atomic operations work, and being able to build primitives based on these atomics. And quite frankly, there be real dragons here (weak memory models are mind-bending), but--like assembly programming--it's not really necessary to resort to this level unless you really need to care about the performance, and the first rule of performance engineering applies [1].

The second level is the level of the standard multithreaded primitives: mutexes, condition variables, semaphores, barriers, etc. This level is important to learn because it's generally the level of principles of how things work that matters most. But it's kind of like a BASIC programming language: it's certainly possible to write in it, but at scale, you get fatigue trying to keep all of the details straight, and it's rapidly apparent that there are lots of common patterns that could be handled more simply.

The highest level of understanding is recognizing that there are a few basic common patterns of multithreading, and if you can just resort to appropriate libraries for those patterns, you generally don't have to worry about the details at all. Chief among these is embarrassingly-parallel code, stuff that generally works with a fork-join model, or generally any sort of pattern that's "for each element in a large set", where the work to be done involves no shared mutable state.

Speaking of which, the most important thing to take away from any multithreaded programming is this: shared mutable state is inherently problematic. If you can avoid having any shared mutable state, that is ideal. The next-best option is to move all of the shared mutable state into a library that someone more competent than you has written (e.g., a database).

[1] Don't optimize until you've measured to be sure you know what you need to optimize.

You can get this kind of behavior in C++ with, e.g., folly::Synchronized.

Imagine writing something with dozens of threads, each running different programs with various threading primitives sprinkled around. You would never be able to debug it, because just conjuring each case of "if this thread gets to here and that thread gets to there" in a test would be impossible.

If you're going to use these, keep it simple.

What I really prefer is not to share the state at all, just have something like a ring buffer between threads that pass messages to each other. Each thread just sits there and takes things from its input buffers and puts things onto the message queues of other threads. The ring buffer itself uses atomics, or is well tested to ensure none of these weird locking scenarios happen. This way all your thinking is just single threaded, and if there's a sync issue you know where to find it.