Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed Systems Shibboleths

jolynch.github.io

Post reply on HN
Print view View original

11–20 of 72 posts

Re: Distributed Systems Shibboleths

#11

Post by jolynch »

rubiquity wrote: post #7

This was a great post and covers many day to day topics that practitioners tend to hand wave over, especially as distributed systems are becoming more pervasive. Dare I say even some of the statements are becoming cliches. The section on distributed transactions could have a little more nuance. Particularly the example about the counter where I suspect any system offering transactions also has a CAS operation. Additi…

Thanks I'm glad you liked it! Your point on distributed transactions is very true, using CAS is what I meant by "transactionally advance a summary".

  For example, you could place a unique identifier on every count event and then roll 
  up those deltas in the background and transactionally advance a summary, either 
  preventing ingestion after some time delay or handling recounting.
Certainly transactions can help, but you still have to data model correctly for failure.

Re: Distributed Systems Shibboleths

#12

Post by User23 »

mjb wrote: post #4

I enjoyed reading that a lot. > The main advantage of distributed transactions is that they make distributed systems look less distributed by choosing CP, but that inherently trades off availability! This is true, but I suspect that its slightly missing the important thing about transactions. A transaction is an operation that takes the database from one Consistent (ACID "C", you can think about it as "legal under th…

I feel like the correct approach is accepting that determinacy is nonsensical in a world where time is relative and instead doubling down on nondeterministic (but predictable!) algorithms. This means leveraging concepts like commutativity and associativity to ensure predictability.

Re: Distributed Systems Shibboleths

#13

Post by jolynch »

andrewflnr wrote: post #6

> database vendors might try just a little harder to tell the truth... Come on, you know that's not what's going to happen. If they notice at all, they'll just incorporate the magic phrases into their BS so you have to hunt harder for a real signal.

Oh shoot I forgot Shibboleths have to remain secret, I have made a terrible mistake.

Re: Distributed Systems Shibboleths

#14

Post by curious_cat_163 »

jolynch wrote: post #13
andrewflnr wrote: post #6

> database vendors might try just a little harder to tell the truth... Come on, you know that's not what's going to happen. If they notice at all, they'll just incorporate the magic phrases into their BS so you have to hunt harder for a real signal.

Oh shoot I forgot Shibboleths have to remain secret, I have made a terrible mistake.

Is there a ‘standard’ way to test the positive shibboleths for existence?

I am not necessarily thinking just tests running as code. Although that would be nice.

Re: Distributed Systems Shibboleths

#15

Post by rubiquity »

jolynch wrote: post #11
rubiquity wrote: post #7

This was a great post and covers many day to day topics that practitioners tend to hand wave over, especially as distributed systems are becoming more pervasive. Dare I say even some of the statements are becoming cliches. The section on distributed transactions could have a little more nuance. Particularly the example about the counter where I suspect any system offering transactions also has a CAS operation. Additi…

Thanks I'm glad you liked it! Your point on distributed transactions is very true, using CAS is what I meant by "transactionally advance a summary". For example, you could place a unique identifier on every count event and then roll up those deltas in the background and transactionally advance a summary, either preventing ingestion after some time delay or handling recounting. Certainly transactions can help, but you…

Thank you for pointing that out. I misinterpreted that bit to mean something less accurate than CAS.

Re: Distributed Systems Shibboleths

#16

Post by jolynch »

mjb wrote: post #4

I enjoyed reading that a lot. > The main advantage of distributed transactions is that they make distributed systems look less distributed by choosing CP, but that inherently trades off availability! This is true, but I suspect that its slightly missing the important thing about transactions. A transaction is an operation that takes the database from one Consistent (ACID "C", you can think about it as "legal under th…

Great points, transactions are certainly useful in helping developers think about state transitions. I think some of the ~snark might come from my personal struggles with trying to convey why wrapping non idempotent state transitions in "BEGIN TRANSACTION ... COMMIT" doesn't immediately make the system reliable. I completely agree transactions make understanding the state transitions easier and that is valuable.

I do think CRDTs or idempotency/fencing tokens are also a valuable way to reason about state transitions, and they can provide much lower latency in a global distributed system.

Re: Distributed Systems Shibboleths

#17

Post by evil-olive »

joebob42 wrote: post #5

Am I missing why a distributed lock is an impossibility? The problem stated is that a partitioned node can't know it has lost the lock, but this is only an issue if there is a way to lose the lock short of returning it. Which I guess is to say: what difference is there between a lease with an infinite timeout unless manually returned, and a "lock"? Certainly the system deadlocks under partition but I'm not sure why t…

> a lease with an infinite timeout unless manually returned

I would argue that "infinite timeout" is another negative shibboleth.

every operation in a distributed system has some duration after which you can be 99.9% confident (or 99.9999%, or whatever threshold you want to pick) that it was lost to the void and will never return a result.

in a robust distributed system, you want to pick a reasonable timeout value, and then take appropriate action in response to the timeout. typically this is retrying the operation, bubbling up a failure message to a higher level, or some combination of the two (retry a few times, fail if all the retries fail).

an infinite timeout represents a deliberate design choice of "I don't want to handle the case of this message or API call being lost in-transit and never returning either success or failure".

in my experience, infinite timeouts are often the cause of "hmm, this thing is up and running but seems 'stuck' and not making any progress, let me try manually restarting this service...OK, that seems to have recovered it" bugs and production alerts.

Re: Distributed Systems Shibboleths

#19

Post by polio »

Beyond the pedantic distinction, is there any real point to not calling "at-least-once delivery with idempotent processing" exactly-once processing? I can't imagine that any external observers would be able to tell.

Re: Distributed Systems Shibboleths

#20

Post by jolynch »

polio wrote: post #19

Beyond the pedantic distinction, is there any real point to not calling "at-least-once delivery with idempotent processing" exactly-once processing? I can't imagine that any external observers would be able to tell.

It conveys a false sense of correctness. Usually the system doing the processing has to use higher level or external methods of providing idempotency.

For example TCP implements "exactly once processing" by your definition but you probably still want Stripe to include idempotency keys in their charge API so you don't pay twice.

Post reply on HN