Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed Systems Shibboleths

jolynch.github.io

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Distributed Systems Shibboleths

#31

Post by choeger »

User23 wrote: post #12
mjb wrote: post #4

I enjoyed reading that a lot. > The main advantage of distributed transactions is that they make distributed systems look less distributed by choosing CP, but that inherently trades off availability! This is true, but I suspect that its slightly missing the important thing about transactions. A transaction is an operation that takes the database from one Consistent (ACID "C", you can think about it as "legal under th…

I feel like the correct approach is accepting that determinacy is nonsensical in a world where time is relative and instead doubling down on nondeterministic (but predictable!) algorithms. This means leveraging concepts like commutativity and associativity to ensure predictability.

> nondeterministic (but predictable!)

Huh? How could a nondeterministic algorithm be predictable? Do you mean algorithms with a nondeterministic but overall irrelevant component ("pick a random element from this set")?

Re: Distributed Systems Shibboleths

#32

Post by jfim »

derefr wrote: post #22
rubiquity wrote: post #10

Earlier quoted context omitted.

The short answer is because if the lock holder fails your other nodes have no way of knowing if the lock holder failed (consequence of FLP Impossibility result). If you set a timeout, then that’s a lease. The long answer is to peel this onion for yourself and see where it leads. It’s a lot of fun.

Why does it matter (in fully-general theory, which is what we're discussing here) if the lock holder fails? The lock is either released, at which point someone else can acquire it; or never released, at which point the system doesn't try to do whatever that lock is about any more. Assuming that every distributed system has to successfully make progress in all cases is just that—an assumption. A design could require t…

I can't come up with a good reason as to why one would want to fail stalled. In what scenario would one want to have a distributed lock that fails in that way?

Re: Distributed Systems Shibboleths

#33

Post by caffeine »

cpitman wrote: post #30

Great post. This one always brings a smile to my face: > Every component is crash-only I was part of the team that developed a distributed, five-9's control system for an industry where downtime costs millions per minute and comes with a federal investigation if long enough. On top of that, the industry is made up of competitors that explicitly distrust each other, so all components had to be truly distributed, with…

> They interpreted crashing as bad quality

The solution here is to rebrand it with some vague euphemism:

“Ah yes the component underwent a state calibration”

Re: Distributed Systems Shibboleths

#34

Post by WJW »

cpitman wrote: post #30

Great post. This one always brings a smile to my face: > Every component is crash-only I was part of the team that developed a distributed, five-9's control system for an industry where downtime costs millions per minute and comes with a federal investigation if long enough. On top of that, the industry is made up of competitors that explicitly distrust each other, so all components had to be truly distributed, with…

Rebrand it as "fault tolerant" and/or "adverse environment certified" and you should be good to go. That's how they do it in the military sector at least.

Re: Distributed Systems Shibboleths

#35

Post by ants_a »

derefr wrote: post #22
rubiquity wrote: post #10

Earlier quoted context omitted.

The short answer is because if the lock holder fails your other nodes have no way of knowing if the lock holder failed (consequence of FLP Impossibility result). If you set a timeout, then that’s a lease. The long answer is to peel this onion for yourself and see where it leads. It’s a lot of fun.

Why does it matter (in fully-general theory, which is what we're discussing here) if the lock holder fails? The lock is either released, at which point someone else can acquire it; or never released, at which point the system doesn't try to do whatever that lock is about any more. Assuming that every distributed system has to successfully make progress in all cases is just that—an assumption. A design could require t…

That's the availability part. If a system is unable to make progress it is not available.

Re: Distributed Systems Shibboleths

#37

Post by fh973 »

cpitman wrote: post #30

Great post. This one always brings a smile to my face: > Every component is crash-only I was part of the team that developed a distributed, five-9's control system for an industry where downtime costs millions per minute and comes with a federal investigation if long enough. On top of that, the industry is made up of competitors that explicitly distrust each other, so all components had to be truly distributed, with…

The term you're looking for is "software rejuvenation".

Jokes aside, there is even a body of research papers around this subject, if you need some backing.

Re: Distributed Systems Shibboleths

#38

Post by __alexs »

cpitman wrote: post #30

Great post. This one always brings a smile to my face: > Every component is crash-only I was part of the team that developed a distributed, five-9's control system for an industry where downtime costs millions per minute and comes with a federal investigation if long enough. On top of that, the industry is made up of competitors that explicitly distrust each other, so all components had to be truly distributed, with…

It's not a crash it's a runtime state rollback.

Re: Distributed Systems Shibboleths

#40

Post by SpicyLemonZest »

polio wrote: post #19

Beyond the pedantic distinction, is there any real point to not calling "at-least-once delivery with idempotent processing" exactly-once processing? I can't imagine that any external observers would be able to tell.

It’s a problem for those of us who build consumer APIs, because people without a deep distributed systems background quite reasonably expect that “exactly once” means their downstream application will receive each event exactly once. I’ve had multiple incredibly frustrating conversations where I had to find a diplomatic way to explain that yes, your connector needs to handle idempotency, and no, it’s not because there’s some “exactly once” technology our system is missing.
Post reply on HN