Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed Locks with Redis (2014)

redis.io

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Distributed Locks with Redis (2014)

#11

Post by hinkley »

alexey-salmin wrote: post #7

Something I don't enjoy about remote/distributed locks is that unlike distributed transactions they're usually unable to provide any strict guarantees about things they protect. E.g. if you algorithm is: 1) Hold the distributed lock 2) Do the thing 3) Release the lock And the node goes dark for a while between steps 1 and 2 (e.g. 100% CPU load), by the time it reaches 2 the lock may have already expired and another n…

If it goes dark a microsecond after #3 you might have an ambiguous success. Transaction processed but you didn't get a confirmation.

A lot of robust systems end up implementing their own bespoke WAL semantics on top of the system of record. It's like we should have a formal solution for doing that by now.

Re: Distributed Locks with Redis (2014)

#12

Post by anonzzzies »

I have seen (ad hoc) implementations go quite bad many times. I encounter them quite a bit as a distributed replacement for some type of db transaction where the db is something like rds; someone thought to be smart and write 'things at scale' (they read on reddit etc) while a db transaction would've been the correct solution and they didn't need scale anyway (or underestimated the current db capabilities for mysql/postgres).

Re: Distributed Locks with Redis (2014)

#13

Post by [deleted] »

anonzzzies wrote: post #12

I have seen (ad hoc) implementations go quite bad many times. I encounter them quite a bit as a distributed replacement for some type of db transaction where the db is something like rds; someone thought to be smart and write 'things at scale' (they read on reddit etc) while a db transaction would've been the correct solution and they didn't need scale anyway (or underestimated the current db capabilities for mysql/p…

[deleted]

Re: Distributed Locks with Redis (2014)

#14

Post by awinter-py »

redis is the easiest-to-host lock server and that's worth the risk in some applications (depending on consequence of errors obv)

inspiring + slightly terrifying that rather than a single server-side implementation, every client is responsible for its own implementation

if postgres provided fast kv cache and a lock primitive it would own

Re: Distributed Locks with Redis (2014)

#15

Post by pram »

zinodaur wrote: post #2

Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

I've personally used Redis locks in my personal projects. The real motivation isn't because it's good or correct but because: Redis is already there in my env.

Re: Distributed Locks with Redis (2014)

#16

Post by pookybear223 »

pram wrote: post #15
zinodaur wrote: post #2

Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

I've personally used Redis locks in my personal projects. The real motivation isn't because it's good or correct but because: Redis is already there in my env.

i think this type of ‘just tacking on’ to projects, since it is so low friction to do so, is part of how we got to where we are today

Re: Distributed Locks with Redis (2014)

#17

Post by gabetax »

pram wrote: post #15
zinodaur wrote: post #2

Martin Kleppmann has some interesting thoughts on Redlock: > I think the Redlock algorithm is a poor choice because it is “neither fish nor fowl”: it is unnecessarily heavyweight and expensive for efficiency-optimization locks, but it is not sufficiently safe for situations in which correctness depends on the lock. https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

I've personally used Redis locks in my personal projects. The real motivation isn't because it's good or correct but because: Redis is already there in my env.

For basic SETNX single instance redis instances, sure.

But for the Redlock algorithm, I've never encountered anyone that was already running 5 redis masters to use out of convenience.

You're much more likely to have an etcd, consul, Zookeeper, etc cluster that you could use for coarse-grained distributed locking.

Re: Distributed Locks with Redis (2014)

#18

Post by pookybear223 »

hinkley wrote: post #8
zinodaur wrote: post #6

Earlier quoted context omitted.

> The algorithm's goal was to move away people that were using a single Redis instance, or a master-slave setup with failover, in order to implement distributed locks, to something much more reliable and safe, but having a very low complexity and good performance. I think this is good perspective. More reliable + more safe + good performance - Fine, its not perfect, but I bet if you are currently using a single node…

Durable locks have a partitioning problem. If the lock holder gets hit by a tornado or catches on fire then there is no recovery method short of manual intervention. I took a formal class on distributed systems back when dinosaurs roamed the earth and the implementation of Ethernet was still considered interesting. And even back then we talked about leases for locks.

time have changed though. there are better implementations of these, and many companies have built successful startups based around these ideas. look around, it’s the age of distributed locking!

Re: Distributed Locks with Redis (2014)

#19

Post by zinodaur »

hinkley wrote: post #8
zinodaur wrote: post #6

Earlier quoted context omitted.

> The algorithm's goal was to move away people that were using a single Redis instance, or a master-slave setup with failover, in order to implement distributed locks, to something much more reliable and safe, but having a very low complexity and good performance. I think this is good perspective. More reliable + more safe + good performance - Fine, its not perfect, but I bet if you are currently using a single node…

Durable locks have a partitioning problem. If the lock holder gets hit by a tornado or catches on fire then there is no recovery method short of manual intervention. I took a formal class on distributed systems back when dinosaurs roamed the earth and the implementation of Ethernet was still considered interesting. And even back then we talked about leases for locks.

We have things we call "durable locks" (but it sounds like thats a loaded term that I don't know the meaning of) that work by recording lock holders in persistent storage + use a corresponding volatile lock when the lock holders need to assert ownership (e.g. to perform a write).

in our system, the only programs that are allowed to take "durable locks" are ones that are guaranteed to complete (ie, their existence is also recorded in persistent storage, and they are retried until completion). The "durable" part means that even if they restart or die, other writers cant jump in and screw things up. The "volatile" part guarantees that only one of them will be writing at the same time.

I wonder what Martin would have to say about our weird little locks

Re: Distributed Locks with Redis (2014)

#20

Post by alexey-salmin »

hinkley wrote: post #11
alexey-salmin wrote: post #7

Something I don't enjoy about remote/distributed locks is that unlike distributed transactions they're usually unable to provide any strict guarantees about things they protect. E.g. if you algorithm is: 1) Hold the distributed lock 2) Do the thing 3) Release the lock And the node goes dark for a while between steps 1 and 2 (e.g. 100% CPU load), by the time it reaches 2 the lock may have already expired and another n…

If it goes dark a microsecond after #3 you might have an ambiguous success. Transaction processed but you didn't get a confirmation. A lot of robust systems end up implementing their own bespoke WAL semantics on top of the system of record. It's like we should have a formal solution for doing that by now.

True. Even simple scenarios like "save a file in s3 IFF the s3 link is saved in postgres" which are seen in virtually any application are rarely handled well.
Post reply on HN