Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed Locks with Redis (2014)

redis.io

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Distributed Locks with Redis (2014)

#31

Post by dang »

cjk wrote: post #4
zinodaur wrote: post #2

Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Salvatore Sanfilippo (author of Redis and Redlock) wrote a response to Martin Kleppmann's analysis that is worth a read (though it is a bit dense and hard to follow at times): http://antirez.com/news/101 I think I agree with Kleppmann's analysis, though.

Discussed at the time:

Is Redlock Safe? Reply to Redlock Analysis - https://news.ycombinator.com/item?id=11065933 - Feb 2016 (135 comments)

Re: Distributed Locks with Redis (2014)

#32

Post by imor80 »

alexey-salmin wrote: post #7

Something I don't enjoy about remote/distributed locks is that unlike distributed transactions they're usually unable to provide any strict guarantees about things they protect. E.g. if you algorithm is: 1) Hold the distributed lock 2) Do the thing 3) Release the lock And the node goes dark for a while between steps 1 and 2 (e.g. 100% CPU load), by the time it reaches 2 the lock may have already expired and another n…

A lot of what you say is explained in detail in Martin Kleppmann's article[0]. As you said, there's no guarantee about when the lock will expire. The proper solution for this is a fencing token. The idea is similar to how people have used optimistic locking when updating data in a db to avoid two users overwriting other's work.

[0]: https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Re: Distributed Locks with Redis (2014)

#33

Post by hinkley »

kijin wrote: post #26
zinodaur wrote: post #19

Earlier quoted context omitted.

We have things we call "durable locks" (but it sounds like thats a loaded term that I don't know the meaning of) that work by recording lock holders in persistent storage + use a corresponding volatile lock when the lock holders need to assert ownership (e.g. to perform a write). in our system, the only programs that are allowed to take "durable locks" are ones that are guaranteed to complete (ie, their existence is…

The thing about most modern web backends is that virtually nothing is guaranteed to complete. The processes that use locks are often short-lived. They live in short-lived containers with no state, or maybe they're just lambdas executing under a strict resource limit. Either way, there's nobody to clean up after them or restart them once they're killed. When they begin a database transaction and then disappear for any…

Autoscaling might mean there isn’t even a machine that corresponds to that dead server for days, weeks, or months.

What GP said sounds like it has leases of a fashion. Maybe not the jargon I’d choose to describe them but the industry is full of misleading names for things.

Re: Distributed Locks with Redis (2014)

#34

Post by wesselbindt »

cjk wrote: post #4
zinodaur wrote: post #2

Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Salvatore Sanfilippo (author of Redis and Redlock) wrote a response to Martin Kleppmann's analysis that is worth a read (though it is a bit dense and hard to follow at times): http://antirez.com/news/101 I think I agree with Kleppmann's analysis, though.

I think this is a good read, but not so much a rebuttal. He never really addresses the following scenario:

1. Get the current time.

2. … All the steps needed to acquire the lock …

3. Get the current time, again.

4. Check if we are already out of time, or if we acquired the lock fast enough.

4.5. client pauses for whatever reason (the example Kleppmann gives is a GC pause), long enough for the lock to expire

4.75. another client acquires the lock

5. Two clients simultaneously hold the lock

Which is the core of Kleppmann's argument against Redlock's correctness. I think the conclusion Sanfilippo can arrive at is that the algorithm is safer than the single node locking algorithm.

Re: Distributed Locks with Redis (2014)

#35

Post by wesselbindt »

zinodaur wrote: post #2

Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

This is a very nice read, as Kleppmann always is. Thanks for the recommendation!

Re: Distributed Locks with Redis (2014)

#36

Post by rand_r »

alexey-salmin wrote: post #7

Something I don't enjoy about remote/distributed locks is that unlike distributed transactions they're usually unable to provide any strict guarantees about things they protect. E.g. if you algorithm is: 1) Hold the distributed lock 2) Do the thing 3) Release the lock And the node goes dark for a while between steps 1 and 2 (e.g. 100% CPU load), by the time it reaches 2 the lock may have already expired and another n…

Yes, exactly! We found out the hard way just how unreliable Redis-based locks are, and switched to Postgres locks. It works reliably since our code is already in a Postgres transaction.

Created a “lock” table with a single string key column, so you can “select key for update” on an arbitrary string key (similar UX to redis lock). I looked at advisory locks, but they don’t work when the lock key needs to be dynamically generated.

Re: Distributed Locks with Redis (2014)

#38

Post by silverwind »

skyde wrote: post #23
awinter-py wrote: post #14

redis is the easiest-to-host lock server and that's worth the risk in some applications (depending on consequence of errors obv) inspiring + slightly terrifying that rather than a single server-side implementation, every client is responsible for its own implementation if postgres provided fast kv cache and a lock primitive it would own

Redis is a very bad store for a distributed lock but Postgres is only slightly better. What you truly need is something like ZooKeeper and etcd that are designed to achieve distributed consensus using algorithms like Paxos or Raft. This ensures strong consistency and reliability in a distributed system, making them ideal for tasks like leader election, configuration management, and lease management where consistency…

Redis can achieve the same with `redis-sentinel` and `min-replicas-to-write`.

Re: Distributed Locks with Redis (2014)

#39

Post by quectophoton »

rand_r wrote: post #36
alexey-salmin wrote: post #7

Something I don't enjoy about remote/distributed locks is that unlike distributed transactions they're usually unable to provide any strict guarantees about things they protect. E.g. if you algorithm is: 1) Hold the distributed lock 2) Do the thing 3) Release the lock And the node goes dark for a while between steps 1 and 2 (e.g. 100% CPU load), by the time it reaches 2 the lock may have already expired and another n…

Yes, exactly! We found out the hard way just how unreliable Redis-based locks are, and switched to Postgres locks. It works reliably since our code is already in a Postgres transaction. Created a “lock” table with a single string key column, so you can “select key for update” on an arbitrary string key (similar UX to redis lock). I looked at advisory locks, but they don’t work when the lock key needs to be dynamicall…

After reading the current[1] top comment about Redlock, this was literally the next low-effort thing that came to mind, so I'm glad to find some else's experiences with using a PostgreSQL table as a lock.

I will need a distributed lock soon, but I've never used one before so I'm taking this chance to learn about them.

[1]: https://news.ycombinator.com/item?id=41315621

Re: Distributed Locks with Redis (2014)

#40

Post by skyde »

silverwind wrote: post #38
skyde wrote: post #23

Earlier quoted context omitted.

Redis is a very bad store for a distributed lock but Postgres is only slightly better. What you truly need is something like ZooKeeper and etcd that are designed to achieve distributed consensus using algorithms like Paxos or Raft. This ensures strong consistency and reliability in a distributed system, making them ideal for tasks like leader election, configuration management, and lease management where consistency…

Redis can achieve the same with `redis-sentinel` and `min-replicas-to-write`.

Redis Sentinel provides high availability and monitoring for Redis, but it does not guarantee strong consistency.

Linearizability requires that once a write is acknowledged, all subsequent reads should reflect that write.

if min-replicas-to-write is set to the number of Redis replica then if a single node goes down you won't be able to do any write (take lock or release lock).

if min-replicas-to-write is set to any number smaller than the total number or Redis replica some replica could still be lagging because of Asynchronous replication.

Also when a replica acknowledges a write in Redis, it means that the write has been received and logged by the replica, but it doesn’t necessarily mean that the write has been fully processed and applied to the data set.

This mean reading from replica that acknowledges a write from master might still return the Old value for the Key.

Post reply on HN