Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed systems programming has stalled

shadaj.me

Post reply on HN
Print view View original

111–120 of 232 posts

Re: Distributed systems programming has stalled

#111

Post by riku_iki »

mgraczyk wrote: post #58

The author is missing information about LLMs. In the "Obligatory LLM Section" he focuses on distributed systems that use LLMs. But almost all of the new innovation I'm familiar with in distributed systems is about training LLMs. I wouldn't say the programming techniques are "new" in the way this post is describing them, but the specifics are pretty different from building a database or data pipeline engine (less mess…

> But almost all of the new innovation I'm familiar with in distributed systems is about training LLMs

I think database space is still hot topic with many unsolved problems.

Re: Distributed systems programming has stalled

#112

Post by AlotOfReading »

Scramblejams wrote: post #37
bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

I've often heard embedded is a nightmare of slapdashery. Any tips for finding shops that do it right?

It's not foolproof, but I've found there's a strong correlation between product margin and the sanity of the dev experience.

Re: Distributed systems programming has stalled

#113

Post by filoleg »

zero_shift wrote: post #30
bsnnkv wrote: post #9

Earlier quoted context omitted.

The real kicker is that it wasn't even management saying this, it was "senior" developers on the team. I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling.

Why would people who are good at [scarce, valuable skill] and get paid [many bananas] to practice it want to even imagine a world where that skill is now redundant? ;-)

The real skill is “problem-solving”, not “doing lots of specific manual steps that could be automated and made easier.”

Unfortunately, some people confuse the two and believe they are paid to do the latter, not the former, simply because others look at those steps and go “wtf, we could make that hell more pleasant and easier to deal with”.

In the same vein, “creating perceived job security for yourself by willing to continuously deal with stupid bs that others rightfully aren’t interested in wasting time on.”

Sadly, you are ultimately right though, as misguided self-interest often tends to win over well-meant proposals.

Re: Distributed systems programming has stalled

#114

Post by tonyarkles »

cmrdporcupine wrote: post #31

Two things: Distributed systems are difficult to reason about. Computer hardware today is very powerful. There is a yo-yo process in our industry over the last 50 years between centralization and distribution. We necessarily distribute when we hit the limits of what centralization can accomplish because in general centralization is easier to reason about. When we hit those junctures, there's a flush of effort into di…

> But then, frankly, mass adoption of SSDs, much more powerful computers, etc. made a lot of those things less necessary. The stuff that most people are doing doesn't require a high level of distributed systems sophistication.

I did my MSc in Distributed Systems and it was always funny (to me) to ask a super simple question when someone was presenting distributed system performance metrics that they'd captured to compare how a system scaled across multiple systems: how long does it take your laptop to process the same dataset? No one ever seemed to have that data.

And then the (in)famous COST paper came out and validated the question I'd been asking for years: https://www.usenix.org/system/files/conference/hotos15/hotos...

Re: Distributed systems programming has stalled

#116

Post by gregw2 »

What I noticed missing in this analysis of distributed systems programming was a recognition/discussion of how distributed databases (or datalakes) decoupling storage from compute have changed the art of the possible.

In the old days of databases, if you put all your data in one place, you could scale up (SMP) but scaling out (MPP) really was challenging. Nowdays, you (iceberg), or a DB vendor (Snowflake, Databricks, BigQuery, even BigTable, etc), put all your data on S3/GCS/ADLS and you can scale out compute to read traffic as much as you want (as long as you accept something like a snapshot isolation read level and traffic is largely read-only or writes are distributed across your tables and not all to one big table.)

You can now share data across your different compute nodes or applications/systems by managing permissions pointers managed via a cloud metadata/catalog service. You can get microservice databases without each having completely separate datastores in a way.

Re: Distributed systems programming has stalled

#117

Post by th0ma5 »

jimbokun wrote: post #90
alabastervlog wrote: post #50

Earlier quoted context omitted.

I've found the rush to distributed computing when it's not strictly necessary kinda baffling. The costs in complexity are extreme. I can't imagine the median company doing this stuff is actually getting either better uptime or performance out of it—sure, it maybe recovers better if something breaks, maybe if you did everything right and regularly test that stuff (approximately nobody does though), but there's also so…

Distributed or not is a very binary function. If you can run in one large server, great, just write everything in non-distributed fashion. But once you need that second server, everything about your application needs to work in distributed fashion.

I wish I could upvote you again. The complexity balloons when you try to adapt something that wasn't distributed, and often things can be way simpler and more robust if you start with a distributed concept.

Re: Distributed systems programming has stalled

#118

Post by achierius »

dekhn wrote: post #102
alabastervlog wrote: post #50

Earlier quoted context omitted.

I've found the rush to distributed computing when it's not strictly necessary kinda baffling. The costs in complexity are extreme. I can't imagine the median company doing this stuff is actually getting either better uptime or performance out of it—sure, it maybe recovers better if something breaks, maybe if you did everything right and regularly test that stuff (approximately nobody does though), but there's also so…

I am always happy when I can take a system that is based on distributed computing, and convert it to a stateless single machine job that runs just as quickly but does not have the complexity associated with distributed computing. Reccently I was going to do a fairly big download of a dataset (45T) and when I first looked at it, figured I could shard the file list and run a bunch of parallel loaders on our cluster. In…

My wheelhouse is lower on the stack, so I'm curious as to what you mean by "stateless single machine job" -- do you just mean that it runs from start to end, without options for suspension/migration/resumption/etc.?

Re: Distributed systems programming has stalled

#120

Post by KaiserPro »

boarush wrote: post #86
sd9 wrote: post #81

Earlier quoted context omitted.

> the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped Gosh, this was hard to parse! I’m still not sure I’ve got it. Do you mean “kubernetes has caused more people to suffer due to going distributed unnecessarily”, or something else?

Had me confused for a second too, but I think it is the former that they meant. K8s has unneeded complexity which is really not required at even decent enough scales, if you've put in enough effort to architect a solution that makes the right calls for your business.

yeah sorry, double negatives.

People got burnt by kubernetes, and that pissed in the well of enthusiasm for experimenting with distributed systems

Post reply on HN