21–30 of 41 posts
Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
Where's the Jepsen suite tests? Without it this is alphaware at best
The only reasonable correctness-oriented view of that is "lol". It's not worth throwing a Jepsen-like test at it, the fundamentals aren't even slightly sound, merely "usually good enough". Whether that's worth it for [use X] depends on that use - often yes!
redis is the easiest-to-host lock server and that's worth the risk in some applications (depending on consequence of errors obv) inspiring + slightly terrifying that rather than a single server-side implementation, every client is responsible for its own implementation if postgres provided fast kv cache and a lock primitive it would own
What you truly need is something like ZooKeeper and etcd that are designed to achieve distributed consensus using algorithms like Paxos or Raft.
This ensures strong consistency and reliability in a distributed system, making them ideal for tasks like leader election, configuration management, and lease management where consistency across nodes is critical.
redis is the easiest-to-host lock server and that's worth the risk in some applications (depending on consequence of errors obv) inspiring + slightly terrifying that rather than a single server-side implementation, every client is responsible for its own implementation if postgres provided fast kv cache and a lock primitive it would own
Redis is a very bad store for a distributed lock but Postgres is only slightly better. What you truly need is something like ZooKeeper and etcd that are designed to achieve distributed consensus using algorithms like Paxos or Raft. This ensures strong consistency and reliability in a distributed system, making them ideal for tasks like leader election, configuration management, and lease management where consistency…
These algorithms ensure that a majority of nodes (a quorum) must agree on any proposed chAnge. This agreement guarantees that once a decision is made (e.g., to commit a transaction), it is final and consistent across all nodes. This strong consistency is critical in distributed systems to avoid split-brain scenarios.
This is easily caused by :
1-network partition
2-latency issues.
3-Async failover (2 nodes think they are the master)
4-replica lag (some but not all replica acknowledged the write) while master send confirmation to client
Earlier quoted context omitted.
If it goes dark a microsecond after #3 you might have an ambiguous success. Transaction processed but you didn't get a confirmation. A lot of robust systems end up implementing their own bespoke WAL semantics on top of the system of record. It's like we should have a formal solution for doing that by now.
True. Even simple scenarios like "save a file in s3 IFF the s3 link is saved in postgres" which are seen in virtually any application are rarely handled well.
1. make sure both operations will be retried if they don't run to completion, and
2. think through how the rest of the system would react to one of them being present without the other
Then I use whichever of the two orderings is less bad from the perspective of #2. Obviously this depends on the exact use case -- I was simply lucky that the rest of the system was designed in such a way that it could tolerate that bad intermediate state.
Earlier quoted context omitted.
Durable locks have a partitioning problem. If the lock holder gets hit by a tornado or catches on fire then there is no recovery method short of manual intervention. I took a formal class on distributed systems back when dinosaurs roamed the earth and the implementation of Ethernet was still considered interesting. And even back then we talked about leases for locks.
We have things we call "durable locks" (but it sounds like thats a loaded term that I don't know the meaning of) that work by recording lock holders in persistent storage + use a corresponding volatile lock when the lock holders need to assert ownership (e.g. to perform a write). in our system, the only programs that are allowed to take "durable locks" are ones that are guaranteed to complete (ie, their existence is…
The processes that use locks are often short-lived. They live in short-lived containers with no state, or maybe they're just lambdas executing under a strict resource limit. Either way, there's nobody to clean up after them or restart them once they're killed. When they begin a database transaction and then disappear for any reason, the best practice is to roll back and pretend they never did anything.
In this brave new world of YOLO lock holders, antirez's position makes a lot of sense. There's definitely still room for old-fashioned durable locks, but these are different use cases.
>The Hacker News user eurleif noticed how it is possible to reacquire the lock as a strategy if the client notices it is taking too much time in order to complete the operation. This can be done by just extending an existing lock, sending a script that extends the expire of the value stored at the key is the expected one. If there are no new partitions, and we try to extend the lock enough in advance so that the keys will not expire, there is the guarantee that the lock will be extended.
Earlier quoted context omitted.
> The algorithm's goal was to move away people that were using a single Redis instance, or a master-slave setup with failover, in order to implement distributed locks, to something much more reliable and safe, but having a very low complexity and good performance. I think this is good perspective. More reliable + more safe + good performance - Fine, its not perfect, but I bet if you are currently using a single node…
Durable locks have a partitioning problem. If the lock holder gets hit by a tornado or catches on fire then there is no recovery method short of manual intervention. I took a formal class on distributed systems back when dinosaurs roamed the earth and the implementation of Ethernet was still considered interesting. And even back then we talked about leases for locks.
Something I don't enjoy about remote/distributed locks is that unlike distributed transactions they're usually unable to provide any strict guarantees about things they protect. E.g. if you algorithm is: 1) Hold the distributed lock 2) Do the thing 3) Release the lock And the node goes dark for a while between steps 1 and 2 (e.g. 100% CPU load), by the time it reaches 2 the lock may have already expired and another n…
If it goes dark a microsecond after #3 you might have an ambiguous success. Transaction processed but you didn't get a confirmation. A lot of robust systems end up implementing their own bespoke WAL semantics on top of the system of record. It's like we should have a formal solution for doing that by now.