Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed systems programming has stalled

shadaj.me

Post reply on HN
Print view View original

61–70 of 232 posts

Re: Distributed systems programming has stalled

#61

Post by hinkley »

shermantanktop wrote: post #39
hinkley wrote: post #28

I am so appalled every time I ask a group of devs and get the same answer that I’ve just stopped asking. How many of you took a distributed programming class in college? And it turns out yet again that not only am I the only one, but that none of them recollect it even being in the course catalog. For me it was a required elective (you must take at least one of these 2-3 classes). And I went to college while web brow…

My .02 is that any topic sufficiently important shouldn't be left to colleges. You can force feed a complex topic to a bunch of undergrads, but they will forget 95% of it, and 5 years later they'll say "ohhh, I think I have a textbook on that in my parents' basement." The reality is that most of this profession is learned on the job, and college acts as a filter at the start of the funnel. If someone is not capable o…

I am 100% convinced we could delete the compiler class from college curricula and replace it with distributed computing and the world would be a better place.

Re: Distributed systems programming has stalled

#62

Post by nine_k »

margorczynski wrote: post #56

Distributed systems are cool but most people don't really get how much complexity it introduces which leads them to fad-driven decisions like using Event Sourcing where there is no fundamental need to use it. I've seen projects getting burned because of the complexity and overhead it introduces where "simpler" approaches worked well and were easy to extend/fix. Hard to find and fix bugs, much slower feature addition…

The best recipe I know is to start from a modular monolith [1] and split it when and if you need to scale way past a few dozen nodes.

Event sourcing is a logical structure; you can implement it with SQLite or even flat files, locally, if you your problem domain is served well by it. Adding Kafka as the first step is most likely a costly overkill.

[1]: https://awesome-architecture.com/modular-monolith/

Re: Distributed systems programming has stalled

#63

Post by [deleted] »

hinkley wrote: post #55

I don’t think there’s anyone in the Elixir community who wouldn’t love it if companies would figure out that everyone is writing software that contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang, and start hiring Elixir or Gleam devs. The future is here, but it is not evenly distributed.

[deleted]

Re: Distributed systems programming has stalled

#64

Post by aDyslecticCrow »

tracnar wrote: post #45

The unison programming language does foray into a truly distributed programming language: https://www.unison-lang.org/

Functional programming languages already have alot of powerful concepts for distributed programming. Loads of the distributed programming techniques used elsewhere are often taken from an obscure fp language from years prior. Erlang comes to mind as still quite uniquely distributed without non fp comparison

Unison seems to build on it further. Very cool

Re: Distributed systems programming has stalled

#65

Post by rjbwork »

margorczynski wrote: post #56

Distributed systems are cool but most people don't really get how much complexity it introduces which leads them to fad-driven decisions like using Event Sourcing where there is no fundamental need to use it. I've seen projects getting burned because of the complexity and overhead it introduces where "simpler" approaches worked well and were easy to extend/fix. Hard to find and fix bugs, much slower feature addition…

If the choice has already been made to do a distributed system (outside of the engineer's control...), is a choice to use Event Sourcing by the engineer then a good idea?

Re: Distributed systems programming has stalled

#66

Post by margorczynski »

nine_k wrote: post #62
margorczynski wrote: post #56

Distributed systems are cool but most people don't really get how much complexity it introduces which leads them to fad-driven decisions like using Event Sourcing where there is no fundamental need to use it. I've seen projects getting burned because of the complexity and overhead it introduces where "simpler" approaches worked well and were easy to extend/fix. Hard to find and fix bugs, much slower feature addition…

The best recipe I know is to start from a modular monolith [1] and split it when and if you need to scale way past a few dozen nodes. Event sourcing is a logical structure; you can implement it with SQLite or even flat files, locally, if you your problem domain is served well by it. Adding Kafka as the first step is most likely a costly overkill. [1]: https://awesome-architecture.com/modular-monolith/

What you're speaking of is a need/usability-based design and extension where you design the solution with certain "safety valves" that let you scale it up when needed.

This is in contrast to the fad-driven design and over-engineering that I'm speaking of (here I simply used ES as an example) that is usually introduced because someone in power saw a blog post or 1h talk and it looked cool. And Kafka will be used because it is the most "scalable" and shiny solution, there is no pros-vs-cons analysis.

Re: Distributed systems programming has stalled

#67

Post by lumost »

bsnnkv wrote: post #2

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level…

Anecdotally, I see a major under appreciation for just how fast and efficient modern hardware is in the distributed systems community.

I’ve seen a great many engineers become so used to provisioning compute that they forget that the same “service” can be deployed in multiple places. Or jump to building an orchestration component when a simple single process job would do the trick.

Re: Distributed systems programming has stalled

#68

Post by spratzt »

cmrdporcupine wrote: post #31

Two things: Distributed systems are difficult to reason about. Computer hardware today is very powerful. There is a yo-yo process in our industry over the last 50 years between centralization and distribution. We necessarily distribute when we hit the limits of what centralization can accomplish because in general centralization is easier to reason about. When we hit those junctures, there's a flush of effort into di…

I would go even further and argue that vast majority of businesses will never need to think about distributed systems. Modern hardware makes them irrelevant to all but the most niche of applications.

Re: Distributed systems programming has stalled

#70

Post by bigmutant »

Good resources for understanding Distributed Systems:

- MIT course with Robert Morris (of Morris Worm fame): https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39...

- Martin Kleppmann (author of DDIA): https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjc...

If you can work through the above (and DDIA), you'll have a solid understanding of the issues in Distributed System, like Consensus, Causality, Split Brain, etc. You'll also gain a critical eye of Cloud Services and be able to articulate their drawbacks (ex: did you know that replication to DynamoDB Secondary Indexes is eventually consistent? What effects can that have on your applications?)

Post reply on HN