Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to do distributed locking

martin.kleppmann.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: How to do distributed locking

#11

Post by iheartmemcache »

mattiemass wrote: post #2

Interesting stuff. In all my distributed systems work so far, I've assumed that a distributed lock is a thing to avoid. I really should take another look at them, just as a tool to have at my disposal.

Oracle has a distributed transaction system that'll work cross-country, but I can't find the details. It's basically a complete rip-off of the OpenVMS implementation from the late 70s (or whenever all those VAX/VMSstations came out - post PDP), no surprises there, and it costs an insane amount just for that functionality (separate from RAC), but it works really well. I'm trying to find the docs since it's been more than half a decade since I used it but [1] is the closest I can get (and that's certainly not it, though RAC does it's job quite well also and that page is worth a read if only for exposing yourself to time-tested transactional models.) It might be HA-NFS over ACFS but that doesn't seem quite right to me. Anyways, read section 5 of this[2] Oracle doc, to have a master-class on maintaining database integrity across the datacenter or across the country. I've used a litany of DBs and I think I wouldn't pick anything made in the last 10 years for my primary store other than Datomic (which I've beaten into the ground as hard as I could and it took my abuse) and or maybeeee VoltDB (if only because Stonebraker is DJB-level smart).

[1] http://docs.oracle.com/cd/B28359_01/server.111/b28318/consis...

Re: How to do distributed locking

#12

Post by bkeroack »

hardwaresofton wrote: post #10

Has anyone used both Zookeeper and etcd in production for management of distributed state? Generally when I think of this problem I reach for etcd, not Zookeeper first, in the hopes of it being lighter (with a relatively vague definition of "light"), and easier to use.

etcd would be a fine choice, but Consul provides a lock feature[1] out of the box.

1. https://www.consul.io/docs/guides/leader-election.html

Re: How to do distributed locking

#13

Post by neeleshs »

hardwaresofton wrote: post #10

Has anyone used both Zookeeper and etcd in production for management of distributed state? Generally when I think of this problem I reach for etcd, not Zookeeper first, in the hopes of it being lighter (with a relatively vague definition of "light"), and easier to use.

Not etcd, but zookeeper yes. It's mostly been set up and forget kind of infrastructure for us, except for some snafus created by our own config errors.

Re: How to do distributed locking

#14

Post by jodah »

mavelikara wrote: post #9
jodah wrote: post #5

Earlier quoted context omitted.

Like anything, it depends on what you're using it for. I wouldn't put a distributed lock into some massively high volume request path, or where absolute availability is required, but it's perfectly fine for some scenarios. As Martin points out though, many (most?) distributed lock implementations are or can be broken in various ways. He hints at one of the fundamental problems - consensus. Many distributed lock imple…

Thanks for posting about Atomix; I had not heard about it before. This seems to be based on Copycat, discussed previously on HN here: https://news.ycombinator.com/item?id=8180360

It is indeed built on Copycat. We're updating the docs right now ahead of an RC, this week or next.

Re: How to do distributed locking

#15

Post by antirez »

Thanks to Martin for analyzing Redlock, I looked forward to an analysis. I don't agree with the two main arguments of the analysis. TLDR: I think the unique random token of Redlock is enough for Check & Set when the lock holder work materializes into a database write (also note that it requires a linearizable database, to check token_A_ID > token_B_ID in the Martin proposed solution), and I think the system model is very real world (AFAIK the algorithm is not dependent on bounded network delays, I think there is an issue in the analysis, details in my blog post so other people can evaluate), and that the fact different processes can count relative time, without any absolute clock, with a bound error (like, 10%) is absolutely credible. However, I'm analyzing the analysis in depth right now and writing a blog post with the details to be posted tomorrow. Also note that usually when you need a distributed lock, it's because you have no sane way to handle race conditions, otherwise you can do with some weaker and/or optimistic locking form to start with.

Re: How to do distributed locking

#16

Post by dvirsky »

dmuth wrote: post #6

It's a long read, but the gist of this article is "use the right tool for the job". Redis is really good for some things, but in its current implementation, distributed locking is not one of them. Along with the author of this blog post, I would recommend the usage of Zookeeper if you have a need for obtaining a lock in a distributed environment. You can read an analysis of how Zookeeper performs under a (intentional…

The author dedicates a good chunk of code to describing a potential problem that might happen in long GC pauses or similar situations. That problem is not unique to redis. Is there any way to avoid it in ZK?

Re: How to do distributed locking

#17

Post by infinite8s »

cryptica wrote: post #8

If you want to have a truly parallel system, you cannot share resources between processes. If you need a lock, then that implies sharing of resources... Which implies limited scalability according to Amdahl's law. To achieve unlimited scalability, you need to make sure that no single data store is shared between all processes and that the amount of interprocess communication (on a per-process basis) doesn't increase…

This. Sometimes it is better to duplicate computation than to wait for shared state (this is in the context of analytics and not a single-source of truth).

Re: How to do distributed locking

#20

Post by teraflop »

dvirsky wrote: post #16
dmuth wrote: post #6

It's a long read, but the gist of this article is "use the right tool for the job". Redis is really good for some things, but in its current implementation, distributed locking is not one of them. Along with the author of this blog post, I would recommend the usage of Zookeeper if you have a need for obtaining a lock in a distributed environment. You can read an analysis of how Zookeeper performs under a (intentional…

The author dedicates a good chunk of code to describing a potential problem that might happen in long GC pauses or similar situations. That problem is not unique to redis. Is there any way to avoid it in ZK?

Yep, and the author even (briefly) describes it. But you can't do it without moving some of the logic into the storage service, to check fencing tokens. Zookeeper exposes various kinds of IDs that can be used to do fencing; apparently (I haven't read the details) Redlock doesn't.

Interestingly, even though HBase uses Zookeeper for coordination, it does not use it for fencing. Instead, fencing is handled by atomic "log-rolling" operations on the HDFS namenode, which can be configured to use its own quorum-based journal. (The equivalent of a "token" is the monotonically-increasing namenode operation sequence number.) So the principle is the same: the system responsible for doing mutual exclusion, and the system that actually stores the data, must be coupled.

Post reply on HN