Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed systems programming has stalled

shadaj.me

Post reply on HN
Print view View original

171–180 of 232 posts

Re: Distributed systems programming has stalled

#171

Post by bob1029 »

DrFalkyn wrote: post #160
bob1029 wrote: post #53

Earlier quoted context omitted.

> Whenever I would bring up this gap I would be told that we can't spend time/money and wait for people to create "magic tools". I've never once been granted explicit permission to try a different path without being burdened by a mountain of constraints that ultimately render the effort pointless. If you want to try a new thing, just build it. No one is going to encourage you to shoot holes through things that they h…

Hope you can justify that during sprint planning / standup

If you are going to just build it in the absence of explicit buy-in, you certainly shouldn't spend time on the standup talking about it. Wait until your idea is completely formed and then drop a 5 minute demo on the team.

It can be challenging to push through to a completed demo without someone cheering you on every morning. I find this to be helpful more than hurtful if we are interested in the greater good. If you want to go against the grain (everyone else on the team), then you need to be really sure before you start wasting everyone else's time. Prove it to yourself first.

Re: Distributed systems programming has stalled

#172

Post by rbjorklin »

intelVISA wrote: post #169
bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

Distributed systems always ends up a dumping ground of failed tech solutions to deep org dysfunction. Weak tech leadership? Let's "fix" that with some microservices. Now it's FUBAR? Conceal it with some cloud native horrors, sacrifice a revolving door of 'smart' disempowered engineers to keep the theater going til you can jump to the next target. Funny because dis sys is pretty solved since Lamport, 40+ years ago.

Would you mind sharing some more specific information/references to Lamport’s work?

Re: Distributed systems programming has stalled

#173

Post by bormaj »

KaiserPro wrote: post #79

Distributed systems are hard, as well all know. However the number of people that actually need a distributed system is pretty small. With the rise of kubernetes, the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped. You go distributed either because you are desperate, or because you think it would be fun. K8s takes the fun out of most things. Moreover, with mac…

Any specific pitfalls to avoid with K8s? I've used it to some degree of success in a production environment, but I keep deployments relatively simple.

Re: Distributed systems programming has stalled

#174

Post by whstl »

intelVISA wrote: post #169
bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

Distributed systems always ends up a dumping ground of failed tech solutions to deep org dysfunction. Weak tech leadership? Let's "fix" that with some microservices. Now it's FUBAR? Conceal it with some cloud native horrors, sacrifice a revolving door of 'smart' disempowered engineers to keep the theater going til you can jump to the next target. Funny because dis sys is pretty solved since Lamport, 40+ years ago.

I suffered through this in two companies and man, it isn't easy.

First one was a multi-billion-Unicorn had everything converted to microservices, with everything customized in Kubernetes. One day I even had to fix a few bugs in the service mesh because the guy who wrote it left and I was the only person not fighting fires able to write the language it was in. I left right after the backend-of-the-frontend failed to sustain traffic during a month where they literally had zero customers (Corona).

At the second one there was a mandate to rewrite everything to microservices and it took another team 5 months to migrate a single 100-line class I wrote into a microservice. It just wasn't meant to be. Then the only guy who knows how the infrastructure works got burnout after being yelled at too many times and then got demoted, and last I heard is at home with depression.

Weak leadership doesn't even begin to describe it, especially the second.

But remembering it is a nice reminder that a job is just a means of getting a payment.

Re: Distributed systems programming has stalled

#175

Post by vitus »

rbjorklin wrote: post #172
intelVISA wrote: post #169

Earlier quoted context omitted.

Distributed systems always ends up a dumping ground of failed tech solutions to deep org dysfunction. Weak tech leadership? Let's "fix" that with some microservices. Now it's FUBAR? Conceal it with some cloud native horrors, sacrifice a revolving door of 'smart' disempowered engineers to keep the theater going til you can jump to the next target. Funny because dis sys is pretty solved since Lamport, 40+ years ago.

Would you mind sharing some more specific information/references to Lamport’s work?

The three big papers: clocks [0], Paxos [1], Byzantine generals [2].

[0] https://lamport.azurewebsites.net/pubs/time-clocks.pdf

[1] https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf

[2] https://lamport.azurewebsites.net/pubs/byz.pdf

Or, if you prefer wiki articles:

https://en.wikipedia.org/wiki/Lamport_timestamp

https://en.wikipedia.org/wiki/Paxos_(computer_science)

https://en.wikipedia.org/wiki/Byzantine_fault

I don't know that I would call it "solved", but he certainly contributed a huge amount to the field.

Re: Distributed systems programming has stalled

#176

Post by madhadron »

rbjorklin wrote: post #172
intelVISA wrote: post #169

Earlier quoted context omitted.

Distributed systems always ends up a dumping ground of failed tech solutions to deep org dysfunction. Weak tech leadership? Let's "fix" that with some microservices. Now it's FUBAR? Conceal it with some cloud native horrors, sacrifice a revolving door of 'smart' disempowered engineers to keep the theater going til you can jump to the next target. Funny because dis sys is pretty solved since Lamport, 40+ years ago.

Would you mind sharing some more specific information/references to Lamport’s work?

Lamport's website has his collected works. The paper to start with is "Time, clocks, and the ordering of events in a distributed system." Read it closely all the way to the end. Everyone seems to miss the last couple sections for some reason.

Re: Distributed systems programming has stalled

#177

Post by sly010 »

bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

I don't disagree, but funny that I recently made a point to someone that modern consumer embedded systems (with multiple MCUs connected with buses and sometimes shared memory) are basically small distributed systems, because partial restarts are common and the start/restart order of the MCUs is not very well defined. At least in the space I am working in. (Needless to say we use C, not rust)

Re: Distributed systems programming has stalled

#178

Post by zootboy »

anitil wrote: post #150
AlotOfReading wrote: post #52

Earlier quoted context omitted.

Most embedded systems are distributed systems these days, there's simply a cultural barrier that prevents most practitioners from fully grappling with that fact. A lot of systems I've worked on have benefited from copying ideas invented by distributed systems folks working on networking stuff 20 years ago.

Yes even 'simple' devices these days will have devices (ADC/SPI etc) running in parallel often using DMA, multiple semi-independent clocks, possibly nested interrupts etc. Oh and the UART for some reason always, always has bugs, so hopefully you're using multiple levels of error checking.

Yeah, it was a "fun" surprise to discover the errata sheet for the microcontroller I was working with after beating my head against the wall trying to figure out why it doesn't do what the reference manual says it should do. It's especially "fun" when the errata is "The hardware flow control doesn't work. Like, at all. Just don't even try."

Re: Distributed systems programming has stalled

#179

Post by szvsw »

zootboy wrote: post #140
AlotOfReading wrote: post #52

Earlier quoted context omitted.

Most embedded systems are distributed systems these days, there's simply a cultural barrier that prevents most practitioners from fully grappling with that fact. A lot of systems I've worked on have benefited from copying ideas invented by distributed systems folks working on networking stuff 20 years ago.

Indeed. I've been building systems that orchestrate batteries and power sources. Turns out, it's a difficult problem to temporally align data points produced by separate components that don't share any sort of common clock source. Just take the latest power supply current reading and subtract the latest battery current reading to get load current? Oops, they don't line up, and now you get bizarre values (like negativ…

I run a small SaaS side hustle where the core value proposition of the product - at least what got us our first customers, even if they did not realize what was happening under the hood - is, essentially, an implementation of NTP running over HTTPS that can be run on some odd devices and sync those devices to mobile phones via a front end app and backend server. There’s some other CMS stuff that makes it easy for the various customers to serve their content to their customers’ devices, but at the end of the day our core trade secret is just using a roll-your-own NTP implementation… I love how NTP is just the tip of the iceberg when it comes to the wicked problem of aligning clocks. This is all just to say - I feel your pain, but also not really since it sounds like you are dealing with higher precision and greater challenges than I ever had to!

Here’s a great podcast on the topic which you will surely like!

https://signalsandthreads.com/clock-synchronization/

And a related HN thread in case you missed it:

https://news.ycombinator.com/item?id=39298652

Re: Distributed systems programming has stalled

#180

Post by fons »

bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

Would you mind disclosing your current employer? I am also interested in moving to an embedded systems role.
Post reply on HN