Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed systems programming has stalled

shadaj.me

Post reply on HN
Print view View original

31–40 of 232 posts

Re: Distributed systems programming has stalled

#31

Post by cmrdporcupine »

Two things:

Distributed systems are difficult to reason about.

Computer hardware today is very powerful.

There is a yo-yo process in our industry over the last 50 years between centralization and distribution. We necessarily distribute when we hit the limits of what centralization can accomplish because in general centralization is easier to reason about.

When we hit those junctures, there's a flush of effort into distributed systems. The last major example of this I can think of was the 2000-2010 period, when MapReduce, "NoSQL" databases, Google's massive arrays of supposedly identical commodity grey boxes (not the case anymore), the High Scalability blog, etc. were the flavour of the time.

But then, frankly, mass adoption of SSDs, much more powerful computers, etc. made a lot of those things less necessary. The stuff that most people are doing doesn't require a high level of distributed systems sophistication.

Distributed systems are an interesting intellectual puzzle. But they should be a means to an end not an end in themselves.

Re: Distributed systems programming has stalled

#32

Post by taeric »

I'm not clear what the proposal here is? It specifically eschews tooling as a driver in the solution, but why? Wouldn't tooling be one of the most likely areas to get solid progress, as you could make tooling and point it at existing products.

Would be interesting to see comparisons to other domains. Surely you could look at things like water processing plants to see how they build and maintain massive structures that do coordinated work between parts of it? Power generation plants. Assembly factories. Do we not have good artifacts for how these things are designed and reasoned about?

Re: Distributed systems programming has stalled

#33

Post by hinkley »

rectang wrote: post #5

Ten years ago, I had lunch with Patricia Shanahan, who worked for Sun on multi-core CPUs several decades ago (before taking a post-career turn volunteering at the ASF which is where I met her). There was a striking similarity between the problems that Sun had been concerned with back then and the problems of the distributed systems that power so much the world today. Some time has passed since then — and yet, most pe…

I think the underlying premise of Cloud is:

Pay a 100% premium on compute resources in order to pretend the 8 Fallacies of Distributed Computing don’t exist.

I sat out the beginning of Cloud and was shocked at how completely absent they are from conversations within the space. When the hangover hits it’ll be ugly. The Devil always gets his due.

Re: Distributed systems programming has stalled

#34

Post by klysm »

herval wrote: post #25

Throwing in my two cents on the LLM impact - I've been seeing an increasing number of systems where core part of the functionality is either LLMs or LLM-generated code (sometimes on the fly, sometimes cached for reuse). If you think distributed systems were difficult before, try to imagine a system where the code being executed _isn't even debuggable or repeatable_. It feels like we're racing towards a level of compl…

That's okay though! We can just make LLMs grasp it!

Re: Distributed systems programming has stalled

#35

Post by shadaj »

gklitt wrote: post #29

This is outside my area of expertise, but the post sounds like it’s asking for “choreographic programming”, where you can write an algorithm in a single function while reasoning explicitly about how it gets distributed: https://en.m.wikipedia.org/wiki/Choreographic_programming I’m curious to what extent the work in that area meets the need.

You caught me! That's what my next post is about :)

Re: Distributed systems programming has stalled

#36

Post by klysm »

hinkley wrote: post #28

I am so appalled every time I ask a group of devs and get the same answer that I’ve just stopped asking. How many of you took a distributed programming class in college? And it turns out yet again that not only am I the only one, but that none of them recollect it even being in the course catalog. For me it was a required elective (you must take at least one of these 2-3 classes). And I went to college while web brow…

Definitely a problem, but I think there's always a gap here. What are colleges optimizing for with computer science programs? I would wager there is an incentive problem at the core which is causing these gaps to occur.

Re: Distributed systems programming has stalled

#37

Post by Scramblejams »

bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

I've often heard embedded is a nightmare of slapdashery. Any tips for finding shops that do it right?

Re: Distributed systems programming has stalled

#38

Post by bongodongobob »

zelphirkalt wrote: post #23
bsnnkv wrote: post #9

Earlier quoted context omitted.

The real kicker is that it wasn't even management saying this, it was "senior" developers on the team. I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling.

Senior doesn't always mean smarter or more experienced or anything really. It just all depends on the company and its culture. It can also mean "worked for longer" (which is not equal to more experienced, as you can famously have 10 times 1y experience, instead of 10y experience) and "more aligned with how management at the company acts".

I'd probably take 10x 1y experience. Where I'm at now, everyone has been with the company 10-40 years. They think the way they do things is the only way because they've never seen anything else. I have many stories similar to the parent. They are a decade behind in their monitoring tooling, if it even exists at all. It's so frustrating when you know there are better ways.

Re: Distributed systems programming has stalled

#39

Post by shermantanktop »

hinkley wrote: post #28

I am so appalled every time I ask a group of devs and get the same answer that I’ve just stopped asking. How many of you took a distributed programming class in college? And it turns out yet again that not only am I the only one, but that none of them recollect it even being in the course catalog. For me it was a required elective (you must take at least one of these 2-3 classes). And I went to college while web brow…

My .02 is that any topic sufficiently important shouldn't be left to colleges. You can force feed a complex topic to a bunch of undergrads, but they will forget 95% of it, and 5 years later they'll say "ohhh, I think I have a textbook on that in my parents' basement."

The reality is that most of this profession is learned on the job, and college acts as a filter at the start of the funnel. If someone is not capable of picking up the Paxos paper, then having had someone tell them about it 5 years ago when they had a hangover won't help.

Post reply on HN