Earlier quoted context omitted.
I think the reason that distributed systems still are the go-to choice for many software teams is to do with people/career expectations/careers orienting themselves around distributed systems over the time period you mentioned. It will take a while for it to re-orient, and then distributed systems might become a fad again ;) An example of this is typical promotion incentives being easier to get in microservice teams,…
Honestly, I am more cynical and just think people are always looking for ways to make their jobs more interesting than they actually are.
Distributed systems programming has stalled
151–160 of 232 posts
Re: Distributed systems programming has stalled
#152Earlier quoted context omitted.
This resonates a lot with me. Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. Embedded systems sometimes require the parallelization of some hot spots, but those are more like the exception AIUI, and you have a lot more control over things; everything is more local…
> Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. I think this take is misguided. Most of the systems nowadays, specially those involving any sort of network cals, are already distributed systems. Yet, the amount of systems go even close to touching fancy consensus…
And Paxos doesn't require much maths. It's pretty tricky to consider all possible interleavings, but in term of maths, it's really basic discrete maths.
Re: Distributed systems programming has stalled
#153Ten years ago, I had lunch with Patricia Shanahan, who worked for Sun on multi-core CPUs several decades ago (before taking a post-career turn volunteering at the ASF which is where I met her). There was a striking similarity between the problems that Sun had been concerned with back then and the problems of the distributed systems that power so much the world today. Some time has passed since then — and yet, most pe…
> and perhaps for good reasons For the very good reason that the underlying math is insanely complicated and tiresome for mere practitioners (which, although I have a background in math, I openly aim to be). For example, even if you assume sequential consistency (which is an expensive assumption) in a C or C++ language multi-threaded program, reasoning about the program isn't easy. And once you consider barriers, ato…
Re: Distributed systems programming has stalled
#154Earlier quoted context omitted.
I've found the rush to distributed computing when it's not strictly necessary kinda baffling. The costs in complexity are extreme. I can't imagine the median company doing this stuff is actually getting either better uptime or performance out of it—sure, it maybe recovers better if something breaks, maybe if you did everything right and regularly test that stuff (approximately nobody does though), but there's also so…
I share your opinions, and really enjoyed your rant. But it's funny. The transition to distributed/cloud feels like the rush to OOP early in my career. All of a sudden there were certain developers who would claim it was impossible to ship features in procedural codebases, and then proceed to make a fucking mess out of everything using classes, completely misunderstanding what they were selling. It is also not unlike…
Re: Distributed systems programming has stalled
#155Earlier quoted context omitted.
This resonates a lot with me. Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. Embedded systems sometimes require the parallelization of some hot spots, but those are more like the exception AIUI, and you have a lot more control over things; everything is more local…
> Even data race free multi-threaded programming in modern C and C++ is incredibly annoying; I dislike dealing with both an explicit mesh of peers, and with a leaky abstraction that lies that threads are "symmetric" (as in SMP) while in reality there's a complicated messaging network underneath. If you're using traditional (p)threads-derived APIs to get work done on a message passing system, I'd say you're using the…
- By "explicit mesh of peers", I referred to atomics, and the modern (C11 and later) memory model. The memory model, for example as written up in the C11 and later standards, is impenetrable. While the atomics interfaces do resemble a messaging passing system between threads, and therefore seem to match the underlying hardware closely, they are discomforting because their foundation, the memory model, is in fact laid out in the PhD dissertation of Mark John Batty, "The C11 and C++11 Concurrency Model" -- 400+ pages! https://www.cl.cam.ac.uk/~pes20/papers/topic.c11.group_abstr...>
- By "leaky abstraction", I mean the stronger posix threads / standard C threads interfaces. They are more intuitive and safer, but are more distant from the hardware, so people sometimes frown at them for being expensive.
Re: Distributed systems programming has stalled
#156Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…
Re: Distributed systems programming has stalled
#157Earlier quoted context omitted.
The real kicker is that it wasn't even management saying this, it was "senior" developers on the team. I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling.
IME, “senior” often means “who is left after the brain-drain & layoffs are done” when you’re at a medium sized company that isn’t prominent.
Re: Distributed systems programming has stalled
#158Earlier quoted context omitted.
_To play devils advocate_: It could've sounded like the "new guy" came in and decided he needed to rewrite everything; bring in new xyx; steer the ship. New guy could even have been stepping directly on the toes of those senior developers who had fought and won wars to get were they are now. In my -very- humble opinion, you should wait at least a year before making big swinging changes or recommendations, most import…
In my less humble opinion: the only honest and objective review you’ll get about a system is from a new hire for about a month. Measure the “what the fucks per hour” as a barometer of how bad your org is and how deep a hole it has dug itself into. After that honeymoon period, all but the most autistic people will learn the organisational politics, keep their head down, and “play the game” to be assigned trivial menia…
FWIW, academia has off-the-charts levels of "wtf" that newcomers will point out, though it's even more ossified than corporate culture, and they don't hire consultants to come in and fix things :)
Re: Distributed systems programming has stalled
#159This is the understatement of the article. There are two insanely difficult things to get right in computers. One is cryptography, and other is distributed systems. I'd argue the latter is harder.
The reason simple enough to understand. In any program the programmer has to carry in his head every piece of state that is accessible at any given point, the invariants that apply to that state, and the code responsible for modifying that state while preserving the invariants. In sequential programs the code that can modify the shared state is restricted to inner loops and functions you call, and you have to verify every modification preserves the invariants. It's a lot. The hidden enemy is aliasing, and you'll find entire books written on the counter measures like immutable objects, function programming, and locks. Coordinating all this is so hard only a small percentage of the population can program large systems. I guess you are thinking "but of a lot of people here can do that". True, but we are a tiny percentage.
In distributed systems those blessed restrictions a single execution thread gives us on what code can access shared state goes out the window. Every line that could read or write the shared state has to be considered, whether its adjacent or not, whether you called it here or not. The state interactions explode in the same way interactions between qubits explode. Both explode beyond the capability of human minds to assemble them all in one place. You have to start forming theorems and formulating proofs.
That worst part is newbie programmers are not usually aware this explosion has taken place. That's why experienced software engineers give the following advice on threads: just don't. You don't have a feel for what will happen, your code will appear to work when you test it while being rabbit warren of disastrous bugs that will likely never be fixed. It's why Linux RCU author Paul McKenney is still not confident his code is correct, despite being one of the greatest concurrent programming minds on the planet. It's why Paxos is hard to understand despite being relatively simple.
Expecting an above average programmer to work on a distributed system and not introduce bugs without leaning on one of one of the "but it is inefficient" tools he lists is an impossible dream. A merely experienced average has no hope. It's hard. Only a tiny, tiny fraction of the programmers on the planet can pull it off kind of hard.
Re: Distributed systems programming has stalled
#160Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…
> Whenever I would bring up this gap I would be told that we can't spend time/money and wait for people to create "magic tools". I've never once been granted explicit permission to try a different path without being burdened by a mountain of constraints that ultimately render the effort pointless. If you want to try a new thing, just build it. No one is going to encourage you to shoot holes through things that they h…