Notes on Distributed Systems for Young Bloods
somethingsimilar.com
Notes on Distributed Systems for Young Bloods
1–10 of 57 posts
Re: Notes on Distributed Systems for Young Bloods
#2Previous discussions from way back:
https://news.ycombinator.com/item?id=5055371
346 points|jcdavis|12 years ago|42 comments
https://news.ycombinator.com/item?id=12245909
386 points|kiyanwang|8 years ago|133 comments
Re: Notes on Distributed Systems for Young Bloods
#3Excellent list; I like the pragmatic and down-to-earth explanations. No buzzwords, no "microservices" (:
I'd say that a good amount of this advice also applies to one-box systems. There can be lots of kinda/sorta distributed sub-components to consider — could be IPC between programs, or even coordination amongst threads in one process. Even the notion of unified memory on one box is a bit of a lie, but at least the hardware can provide some better guarantees than you get in "real" distributed cases.
A lot of the advice where they compare "distributed" to "single-machine" could pretty well apply to "multi-threaded" vs "single-threaded," too.
And on another axis, once you make a program and give it to various people to run, it becomes sort of a "distributed" situation, too — now you have to worry about different versions of that program existing in the wild, compatibility between them and upgrade issues, etc. So things like feature flags, mentioned in the article, can be relevant there, as well.
It's perhaps more of a spectrum of distributedness: from single-CPU to multi-CPU, to multi-computer-tightly-connected, to multi-computer-globally-distributed, with various points in between. And multiple dimensions.
Re: Notes on Distributed Systems for Young Bloods
#4One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.
Re: Notes on Distributed Systems for Young Bloods
#5I share this doc with the most promising people I get to work with.
When I worked at Lookout, Jeff Hodges shared this essay as a presentation, and ended it with a corollary: don't pretend that engineering isn't political. People that think that the code speaks for itself are missing out on important aspects of how to influence the way things are built and how to truly get results.
Ten years later, and there are few people who still so concisely understand the intersection of engineering leadership and those table-stakes capabilities we normally associate with SRE / DevOps.
Re: Notes on Distributed Systems for Young Bloods
#6One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.
Yes but in practice this is not a problem because the bits that are impossible are so narrow that turning at-least-once into exactly-once is so easy it's a service offered by cloud vendors https://cloud.google.com/pubsub/docs/exactly-once-delivery
Re: Notes on Distributed Systems for Young Bloods
#7One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.
Yes but in practice this is not a problem because the bits that are impossible are so narrow that turning at-least-once into exactly-once is so easy it's a service offered by cloud vendors https://cloud.google.com/pubsub/docs/exactly-once-delivery
Those only work because they have retries built in at a layer that runs inside your service. You should understand this because it can have implications for the performance of your system during failures.
Re: Notes on Distributed Systems for Young Bloods
#8One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.
Yes but in practice this is not a problem because the bits that are impossible are so narrow that turning at-least-once into exactly-once is so easy it's a service offered by cloud vendors https://cloud.google.com/pubsub/docs/exactly-once-delivery
"Redelivery versus duplicate" is doing quite a lot of work in there. This is an "at least once" delivery system providing building blocks that you can use to cope with the fact that it's physically impossible to prevent redelivery under some circumstances, which are not actually that rare because some of those circumstances are your fault, not Google's, etc.
Re: Notes on Distributed Systems for Young Bloods
#9I had the pleasure to briefly work with the author of this post within the last few years. Jeff was one of the most enlightening and positive people I've ever learned from. He was refreshingly honest about what challenges he was having, and delightfully accessible for mentorship and advice.
Re: Notes on Distributed Systems for Young Bloods
#10One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.
Yes but in practice this is not a problem because the bits that are impossible are so narrow that turning at-least-once into exactly-once is so easy it's a service offered by cloud vendors https://cloud.google.com/pubsub/docs/exactly-once-delivery
My experience is building streaming systems using “exactly-once delivery” primitives is much more awkward than designing your system around at least once primitives and explicitly de-duplicating using last-write wins. For one thing, LWW gives you an obvious recovery strategy if you have outages of the primitive your system is built on: a lot of the exactly once modes for tools make failure recovery harder than necessary