Live data from Hacker News

A proposal for more reliable locks using Redis

antirez.com

21–27 of 27 posts

Re: A proposal for more reliable locks using Redis

#21
Is this new? I feel like using Redis for locks is something that's been going around for a while. I've used Redis to make locks, and also used it to make counting semaphores. It's a fairly interesting use because it's frequently the the simplest, least overhead means to solve a problem that's reliable enough without actually being reliable.

The most obvious issue is that if Redis goes down you could end up with problems if the processes using the locks continue, particularly depending on when and what state Redis restarts.

Another is that you have to take an approach of re-checking out your lock so as not to let it expire if you can't guarantee strict time constraints. Once you do this, you run a risk of something not finishing but extending its lock indefinitely.

A final issue is that you can end up with a situation where there's no guarantee that a waiting task (or whatever you call something that wants a lock or in on a semaphore) will ever run.

I don't really buy those who talk about this being an insane violation 0 state/share nothing. When I've needed these kinds of primitives it rarely has to do with the application state itself - for example I've used the counting semaphores to control how many worker processes can be active. Likewise, I've used the plain locks (and lock-like structures) to do things like insure atomic/ordered writes for user sessions (I suppose session is stateful, but it's also not really shared application state).

In any case, there are some issues, but at the cost of a minimal amount of nursing the ease of implementation and integration often makes Redis a go-to choice for these kinds of things, particularly in resource (hardware) constrained environments. On the other hand if you're operating a scale where you've got multiple datacenters and such, it's a different ballgame.

Re: A proposal for more reliable locks using Redis

#22

Is this new? I feel like using Redis for locks is something that's been going around for a while. I've used Redis to make locks, and also used it to make counting semaphores. It's a fairly interesting use because it's frequently the the simplest, least overhead means to solve a problem that's reliable enough without actually being reliable. The most obvious issue is that if Redis goes down you could end up with probl…

It is not new at all, that's exactly why I wrote the blog post, since it is an actual use case. But it is a good idea to formalize a canonical model to do it with Redis in a safe way, and to show what are the failure modes when we resort to simpler designs (like failover to slaves).

Re: A proposal for more reliable locks using Redis

#23
post #22

Is this new? I feel like using Redis for locks is something that's been going around for a while. I've used Redis to make locks, and also used it to make counting semaphores. It's a fairly interesting use because it's frequently the the simplest, least overhead means to solve a problem that's reliable enough without actually being reliable. The most obvious issue is that if Redis goes down you could end up with probl…

It is not new at all, that's exactly why I wrote the blog post, since it is an actual use case. But it is a good idea to formalize a canonical model to do it with Redis in a safe way, and to show what are the failure modes when we resort to simpler designs (like failover to slaves).

As a related aside, I'd be mildly interested in Redis having an ability to produce and produce and set UUIDs. That sort of thing might also have some utility for locking structures/synchronization primitives, though it's not the original reason I'd thought having something like that might be nice.

Re: A proposal for more reliable locks using Redis

#24

I wrote a very similar, Redis-based lock in Python a while ago, here it is: https://gist.github.com/adewes/6103220 It uses Redis pipelines and watchers to make sure that no race conditions between two processes requiring the same lock occur, and uses "expire" keys to avoid deadlocks.

You should just use redis-py's Lock (https://github.com/andymccurdy/redis-py/blob/master/redis/cl...)

Re: A proposal for more reliable locks using Redis

#25

I wrote a very similar, Redis-based lock in Python a while ago, here it is: https://gist.github.com/adewes/6103220 It uses Redis pipelines and watchers to make sure that no race conditions between two processes requiring the same lock occur, and uses "expire" keys to avoid deadlocks.

I prefer Lua-scripting based approach: https://github.com/bbangert/retools/blob/master/retools/lock..., but I had to implement something similar to your code to support older Redis versions.

Re: A proposal for more reliable locks using Redis

#26
post #6

Earlier quoted context omitted.

It seems like a timeout is less reliable/safe than some broadcast/ping mechanism that can check availability perpetually and if a node has disappeared the validity of the lock changes. Trying to remember which distributed system model it is that sort of does this. Ring? Mesh?

That could increase throughput if you have a lot of crashing nodes but how does it improve safety or reliability?

Safety? It doesn't. Reliability? because it would prevent another node from acquiring a held lock if a server is available and release a lock if a server goes down.

There would never be a dirty read possible.

Re: A proposal for more reliable locks using Redis

#27
post #12

I finish reading after first paragraph... When do people will learn that using locks, shared memory does not work? It is just wrong. Things should be immutable, you should share nothing. And by nothing i mean nothing at all. It is just WRONG.

Go create such a system, make it easy to use and widely available, and then you might have room to say such things with some credibility.
Post reply on HN