Great post!
A few more positive shiboleths.
One would be eventual consistency.
Another would be discussing write paths vs read paths (or patterns) and recognizing that those can be decoupled (or a mention of CQRS).
Distributed Systems Shibboleths
21–30 of 72 posts
Re: Distributed Systems Shibboleths
#22Am I missing why a distributed lock is an impossibility? The problem stated is that a partitioned node can't know it has lost the lock, but this is only an issue if there is a way to lose the lock short of returning it. Which I guess is to say: what difference is there between a lease with an infinite timeout unless manually returned, and a "lock"? Certainly the system deadlocks under partition but I'm not sure why t…
The short answer is because if the lock holder fails your other nodes have no way of knowing if the lock holder failed (consequence of FLP Impossibility result). If you set a timeout, then that’s a lease. The long answer is to peel this onion for yourself and see where it leads. It’s a lot of fun.
Why does it matter (in fully-general theory, which is what we're discussing here) if the lock holder fails? The lock is either released, at which point someone else can acquire it; or never released, at which point the system doesn't try to do whatever that lock is about any more. Assuming that every distributed system has to successfully make progress in all cases is just that—an assumption. A design could require that operations "fail stalled." Like a physical padlock that is "tamper proof" in the sense that it permanently seizes up when you put in the wrong key.
Re: Distributed Systems Shibboleths
#23Beyond the pedantic distinction, is there any real point to not calling "at-least-once delivery with idempotent processing" exactly-once processing? I can't imagine that any external observers would be able to tell.
[deleted]
Re: Distributed Systems Shibboleths
#24Great post! A few more positive shiboleths. One would be eventual consistency. Another would be discussing write paths vs read paths (or patterns) and recognizing that those can be decoupled (or a mention of CQRS).
> One would be eventual consistency.
In the vein of the TFA, this would be a negative shibboleth (which just goes to show how being pedantic in certain contexts is just silly). The reason being that eventual consistency has no guarantee on what "eventual" means. If your replicas converge ten years from when a change is made, you can (correctly) claim to have eventual consistency.
Re: Distributed Systems Shibboleths
#25For those curious, here's the origin of the word "Shibboleth"
https://www.sefaria.org/Judges.12.6?ven=Tanakh:_The_Holy_Scr...
Re: Distributed Systems Shibboleths
#26Am I missing why a distributed lock is an impossibility? The problem stated is that a partitioned node can't know it has lost the lock, but this is only an issue if there is a way to lose the lock short of returning it. Which I guess is to say: what difference is there between a lease with an infinite timeout unless manually returned, and a "lock"? Certainly the system deadlocks under partition but I'm not sure why t…
> a lease with an infinite timeout unless manually returned I would argue that "infinite timeout" is another negative shibboleth. every operation in a distributed system has some duration after which you can be 99.9% confident (or 99.9999%, or whatever threshold you want to pick) that it was lost to the void and will never return a result. in a robust distributed system, you want to pick a reasonable timeout value, a…
I agree; it's also one of the few issues I have with Linux (and possibly Unix generally).
Zombie processes (dependent on some lock that will never clear) shouldn't be possible. At the very least abort (kill -9) should always be possible.
Failure should always be an option; it should be the default assumption. All other order must be wrested from that chaos.
Re: Distributed Systems Shibboleths
#27Earlier quoted context omitted.
The short answer is because if the lock holder fails your other nodes have no way of knowing if the lock holder failed (consequence of FLP Impossibility result). If you set a timeout, then that’s a lease. The long answer is to peel this onion for yourself and see where it leads. It’s a lot of fun.
Why does it matter (in fully-general theory, which is what we're discussing here) if the lock holder fails? The lock is either released, at which point someone else can acquire it; or never released, at which point the system doesn't try to do whatever that lock is about any more. Assuming that every distributed system has to successfully make progress in all cases is just that—an assumption. A design could require t…
Because you cannot differentiate a slow node from a dead node. People expect different responses to these.
Re: Distributed Systems Shibboleths
#28Earlier quoted context omitted.
Oh shoot I forgot Shibboleths have to remain secret, I have made a terrible mistake.
Is there a ‘standard’ way to test the positive shibboleths for existence? I am not necessarily thinking just tests running as code. Although that would be nice.
I think it's basically pay Aphyr/Jepsen $mondoConsultingRates. :)
Re: Distributed Systems Shibboleths
#29Earlier quoted context omitted.
Why does it matter (in fully-general theory, which is what we're discussing here) if the lock holder fails? The lock is either released, at which point someone else can acquire it; or never released, at which point the system doesn't try to do whatever that lock is about any more. Assuming that every distributed system has to successfully make progress in all cases is just that—an assumption. A design could require t…
Because you cannot differentiate a slow node from a dead node. People expect different responses to these.
Big assumption that a distributed system has to serve “people” and have “responses.”
A distributed system might be, for example, the ACH system: all batch, all store-and-forward, no replies flowing down the line, only processed message response batches dropped off “eventually” in outboxes.
Or, for another example: any workload manager, whether on an HPC cluster or your local Kubernetes node. No synchronous workload execution; just async batch scheduler enqueue with later best-effort status querying.
Re: Distributed Systems Shibboleths
#30Great post. This one always brings a smile to my face:
> Every component is crash-only
I was part of the team that developed a distributed, five-9's control system for an industry where downtime costs millions per minute and comes with a federal investigation if long enough. On top of that, the industry is made up of competitors that explicitly distrust each other, so all components had to be truly distributed, with no central coordination for anything.
Given the requirements we decided to explicitly adopt a crash-only approach. Between idempotent operations, horizontal scaling, and fast restart times, we could make failing components not impact SLAs (and we had testing to ensure it).
Once it gets out into the field (which because of how risk adverse this industry is, is measured in years), it turns out they really did not like software crashing. They interpreted crashing as bad quality, and no amount of "we do it on purpose to ensure correctness" was going to make them happy.