51–60 of 72 posts
Follow the guidelines in this post and you'll indeed result in (more) robust systems. Great writeup.
Am I missing why a distributed lock is an impossibility? The problem stated is that a partitioned node can't know it has lost the lock, but this is only an issue if there is a way to lose the lock short of returning it. Which I guess is to say: what difference is there between a lease with an infinite timeout unless manually returned, and a "lock"? Certainly the system deadlocks under partition but I'm not sure why t…
> a lease with an infinite timeout unless manually returned I would argue that "infinite timeout" is another negative shibboleth. every operation in a distributed system has some duration after which you can be 99.9% confident (or 99.9999%, or whatever threshold you want to pick) that it was lost to the void and will never return a result. in a robust distributed system, you want to pick a reasonable timeout value, a…
Earlier quoted context omitted.
At our job someone decided to use a ready-to-use Go library which used Redis for distributed locking. But I found that it was broken by design and completely unreliable, and we had random transient errors stemming from it. It worked OK 99.9% the time, but once in a while we were getting inconsistent state in our application. The description initially made sense and the usage looked simple. It worked by a node creatin…
So Redis isn't really a distributed locking system. The locks are all managed by a central, non-distributed server: Redis. But this kind of lock is useful too. One nice approach to handling crashes in a system like this is to use the idea of fate sharing [1]: you upper-bound the lifetime of a held lock by the lifetime of the TCP connection it was taken on. When the connection goes, the lock is auto-released. To suppo…
Earlier quoted context omitted.
So Redis isn't really a distributed locking system. The locks are all managed by a central, non-distributed server: Redis. But this kind of lock is useful too. One nice approach to handling crashes in a system like this is to use the idea of fate sharing [1]: you upper-bound the lifetime of a held lock by the lifetime of the TCP connection it was taken on. When the connection goes, the lock is auto-released. To suppo…
Redis can be distributed FWIW, it has two clustering modes. It's just that sharded distribution comes with a bunch of caveats.
Earlier quoted context omitted.
Redis can be distributed FWIW, it has two clustering modes. It's just that sharded distribution comes with a bunch of caveats.
Yes, absolutely! The caveats are relevant to distributed locking , though: sharding would help scale out a locking system horizontally, but each subset of keyspace would still be a non-distributed locking service. Primary-secondary replication doesn't (as far as I can tell!) offer the necessary invariants to act as a locking service - at least, not when employing the straightforward technique GP mentions.
Am I missing why a distributed lock is an impossibility? The problem stated is that a partitioned node can't know it has lost the lock, but this is only an issue if there is a way to lose the lock short of returning it. Which I guess is to say: what difference is there between a lease with an infinite timeout unless manually returned, and a "lock"? Certainly the system deadlocks under partition but I'm not sure why t…
> a lease with an infinite timeout unless manually returned I would argue that "infinite timeout" is another negative shibboleth. every operation in a distributed system has some duration after which you can be 99.9% confident (or 99.9999%, or whatever threshold you want to pick) that it was lost to the void and will never return a result. in a robust distributed system, you want to pick a reasonable timeout value, a…
[1] https://codahale.com/you-cant-sacrifice-partition-tolerance/
I prefer the term retryable to idempotent. If there's a failure in the first call, to be truly idempotent it should fail on the second.
Retryable on the other hand is easier to argue about. Important thing is not the response but the end state of the system.
I liked the article a lot. I prefer the term retryable to idempotent. If there's a failure in the first call, to be truly idempotent it should fail on the second. Retryable on the other hand is easier to argue about. Important thing is not the response but the end state of the system.
Alternatively, idempotency applies to successful operations, orthogonal from error cases.
Retryability doesn't help in the case of the (bad) operation "+1" which is not idempotent.
Earlier quoted context omitted.
I feel like the correct approach is accepting that determinacy is nonsensical in a world where time is relative and instead doubling down on nondeterministic (but predictable!) algorithms. This means leveraging concepts like commutativity and associativity to ensure predictability.
> nondeterministic (but predictable!) Huh? How could a nondeterministic algorithm be predictable? Do you mean algorithms with a nondeterministic but overall irrelevant component ("pick a random element from this set")?
Example: size of a set = 1 + (remove an element from the set and then compute size of reduced set, or 0 if set is empty)