Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed systems programming has stalled

shadaj.me

Post reply on HN
Print view View original

71–80 of 232 posts

Re: Distributed systems programming has stalled

#71

Post by toast0 »

EtCepeyd wrote: post #12
bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

This resonates a lot with me. Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. Embedded systems sometimes require the parallelization of some hot spots, but those are more like the exception AIUI, and you have a lot more control over things; everything is more local…

That's true, but you can do a lot of that once, and then get on with your life, if you build the right structures. I've gotten a huge amount of mileage from consensus to decide where to send reads/writes to, then everyone sends their reads/writes for the same piece of data to the same place; that place does the application logic where it's simple, and sends the result back. If you don't get the result back in time, bubble it up to the end-user application and it may retry or not, depending.

This is built upon a framework of the network is either working or the server team / ops team is paged and will be actively trying to figure it out. It doesn't work nearly as well if you work in an environment where the network is consistently slightly broken.

Re: Distributed systems programming has stalled

#72

Post by bigmutant »

rectang wrote: post #5

Ten years ago, I had lunch with Patricia Shanahan, who worked for Sun on multi-core CPUs several decades ago (before taking a post-career turn volunteering at the ASF which is where I met her). There was a striking similarity between the problems that Sun had been concerned with back then and the problems of the distributed systems that power so much the world today. Some time has passed since then — and yet, most pe…

The fundamental problems are communication lag and lack of information about why issues occur (encapsulated by the Byzantine Generals problem). I like to imagine trying to build a fault-tolerant, reliable system for the Solar System. Would the techniques we use today (retries, timeouts, etc) really be adequate given that lag is upwards of hours instead of milliseconds? But that's the crux of these systems, coordination (mostly) works because systems are close together (same board, at most same DC)

Re: Distributed systems programming has stalled

#73

Post by Karrot_Kream »

When I was graduating from my Masters (failed PhD :) this overview of various programming models is generally how I thought of things.

I've been writing distributed code now in industry for a long time and in practice, having worked at a some pretty high-scale tech companies over the years, most shops tend to favor static-location style models. As the post states, it's due largely to control and performance. Scaling external-distribution systems has been difficult everywhere I've seen it tried and usually ends up creating a few knowledgeable owners of a system with high bus-factor. Scaling tends to work fine until it doesn't and these discontinuous, sharp edges are very very painful as they're hard to predict and allocate resourcing for.

Are external-distribution systems dead ends then? Even if they can achieve high theoretical performance, operation of these systems tends to be very difficult. Another problem I find with external-distribution systems is that there's a lot of hidden complexity in just connecting, reading, and writing to them. So you want to talk to a distributed relational DB, okay, but are you using a threaded concurrency model or an async concurrency model? You probably want a connection pool so that TCP HOL blocking doesn't tank your throughput. But if you're using threads, how do you map your threads to the connections in the pool? The pool itself represents a bottleneck as well. How do you monitor the status of this pool? Tools like Istio strive to standardize this a little bit but fundamentally we're working with 3 domains here just to write to the external-distribution system itself: the runtime/language's concurrency model, the underlying RPC stack, and the ingress point for the external-distribution system.

Does anyone have strong stories of scaling an external-distribution system that worked well? I'd be very curious. I agree that progress here has stalled significantly. But I find myself designing big distributed architecture after big distributed architecture continuing to use my deep experience of architecting these systems to build static-location systems because if I'm already dealing with scaling pains and cross-domain concerns, I may as well rip off the band-aid and be explicit about crossing execution domains.

Re: Distributed systems programming has stalled

#74

Post by [deleted] »

dingnuts wrote: post #3

>The static-location model seems like the right place to start, since it is at least capable of expressing all the types of distributed systems we might want to implement, even if the programming model offers us little help in reasoning about the distribution. We were missing two things that the arbitrary-location model offered: > Writing logic that spans several machines right next to each other, in a single functio…

[deleted]

Re: Distributed systems programming has stalled

#75

Post by ikety »

hinkley wrote: post #55

I don’t think there’s anyone in the Elixir community who wouldn’t love it if companies would figure out that everyone is writing software that contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang, and start hiring Elixir or Gleam devs. The future is here, but it is not evenly distributed.

It's so odd seeing people dissuade others for implementing "niche" languages like Elixir or Gleam. If you post a job opportunity with these languages, I guarantee you will be swamped with qualified candidates that are very passionate and excited to work with these languages full time.

Re: Distributed systems programming has stalled

#76

Post by Karrot_Kream »

sanity wrote: post #59

The article makes great points about why distributed programming has stalled, but I think there's still room for innovation—especially in how we handle state consistency in decentralized systems. In Freenet[1], we’ve been exploring a novel approach to consistency that avoids the usual trade-offs between strong consistency and availability. Instead of treating state as a single evolving object, we model updates as sum…

Haven't read the post yet (I should, I have been vaguely following y'all along but obviously not close enough!) How is this different from delta-based CRDTs? I've built (admittedly toy) CRDTs as DAGs that ship deltas using lattice operations and it's really not that hard to have it work. There's already CRDT based distributed stores out there. How is this any different?

Re: Distributed systems programming has stalled

#77

Post by ignoramous »

bigmutant wrote: post #70

Good resources for understanding Distributed Systems: - MIT course with Robert Morris (of Morris Worm fame): https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39... - Martin Kleppmann (author of DDIA): https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjc... If you can work through the above (and DDIA), you'll have a solid understanding of the issues in Distributed System, like Consensus, Causality, Sp…

> Robert Morris (of Morris Worm fame)

(of Y Combinator fame, too)

Re: Distributed systems programming has stalled

#78

Post by DanielHB »

AlotOfReading wrote: post #52
EtCepeyd wrote: post #12

Earlier quoted context omitted.

This resonates a lot with me. Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. Embedded systems sometimes require the parallelization of some hot spots, but those are more like the exception AIUI, and you have a lot more control over things; everything is more local…

Most embedded systems are distributed systems these days, there's simply a cultural barrier that prevents most practitioners from fully grappling with that fact. A lot of systems I've worked on have benefited from copying ideas invented by distributed systems folks working on networking stuff 20 years ago.

I worked in an IoT platform that consisted of 3 embedded CPUs and one linux board. The kicker was that the linux board could only talk directly to one of the chips, but had to be capable of updating the software running on all of them.

That platform was parallelizable of up to 6 of its kind in a master-slave configuration (so the platform in the physical position 1 would assume the "master role" for a total of 18 embedded chips and 6 linux boards) on top of having optionally one more box with one more CPU in it for managing some other stuff and integrating with each of our clients hardware. Each client had a different integration, but at least they mostly integrated with us, not the other way around.

Yeah it was MUCH more complex than your average cloud. Of course the original designers didn't even bother to make a common network protocol for the messages, so each point of communication not only used a different binary format, they also used different wire formats (CAN bus, Modbus and ethernet).

But at least you didn't need to know kubernetes, just a bunch of custom stuff that wasn't well documented. Oh yeah and don't forget the boot loaders for each embedded CPU, we had to update the bootloaders so many times...

The only saving grace is that a lot of the system could rely on the literal physical security because you need to have physical access (and a crane) to reach most of the system. Pretty much only the linux boards had to have high security standards and that was not that complicated to lock down (besides maintaining a custom yocto distribution that is).

Re: Distributed systems programming has stalled

#79

Post by KaiserPro »

Distributed systems are hard, as well all know.

However the number of people that actually need a distributed system is pretty small. With the rise of kubernetes, the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped.

You go distributed either because you are desperate, or because you think it would be fun. K8s takes the fun out of most things.

Moreover, with machines suddenly getting vast IO improvements, the need for going distributed is much less than it was 10 years. (yes i know there is fault tolerance, but that adds another dimension of pain.)

Re: Distributed systems programming has stalled

#80

Post by moffkalast »

klysm wrote: post #34
herval wrote: post #25

Throwing in my two cents on the LLM impact - I've been seeing an increasing number of systems where core part of the functionality is either LLMs or LLM-generated code (sometimes on the fly, sometimes cached for reuse). If you think distributed systems were difficult before, try to imagine a system where the code being executed _isn't even debuggable or repeatable_. It feels like we're racing towards a level of compl…

That's okay though! We can just make LLMs grasp it!

"There's always a larger model."
Post reply on HN