Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to do distributed locking (2016)

martin.kleppmann.com

Post reply on HN
Print view View original

81–90 of 99 posts

Re: How to do distributed locking (2016)

#81

Post by wbl »

dataflow wrote: post #66
winwang wrote: post #63

Earlier quoted context omitted.

What if the rack goes down? But I think the author is saying a similar thing to you. The fenced token is essentially asserting that the client will no longer rely on the lock, even if it tries to. The difference is the service doesn't need any acknowledgement, no permission needed to simlly deny the client later.

To be clear, my objection is to the premise, not to the offered solution. To your question, could you clarify what exactly you mean by the rack "going down"? This encompasses a lot of different scenarios, I'm not sure which one you're asking about. The obvious interpretation would break all the connections the program has to the outside world, thus preventing the problem by construction.

The process that owns the lock is never heard from again.

Re: How to do distributed locking (2016)

#82

Post by dataflow »

winwang wrote: post #80
dataflow wrote: post #75

Earlier quoted context omitted.

In that scenario the machine would become aware that it can't reach the storage service either, no? In which case the host can terminate the program, or the network can break all the connections between them, or whatever. By default I would think that the lease shouldn't be broken until the network partition gets resolved, but I think the storage system could have a timeout for breaking the lease in that scenario if…

Everything you're saying is plausibly possible in the absurdly large search space of all possible scenarios. The author's premise, however, is rooted in the specific scenario they lay out, with historical supporting examples which you can look into. Even then, the premise before all that was essentially: Redlock does not do what people might expect of a distributed lock. Btw I do have responses to your questions, but…

> The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly".

I get that -- and honestly, I'm not expecting a treatise on distributed consensus here. But what took me aback was that the blog post didn't even attempt to mention anything about the fact that the premise (at first glance) looks glaringly broken. If he'd even said 1 single sentence like "it's {difficult/infeasible/impossible} to design a client that will never continue execution past a timeout", it'd have been fine, and I would've happily moved along. But the way it is written right now, it reads a little bit like: "we design a ticking time bomb that we can't turn off; how can we make sure we don't forget to reset the timer every time?"... without bothering to say anything about why we should be digging ourselves into such a hole in the first place.

Re: How to do distributed locking (2016)

#83

Post by Quekid5 »

eknkc wrote: post #25

I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something. Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some…

Advisory locks have many pitfalls, see [0].

AFAIK the only correct way to do what you probably thought you were doing is "EXCLUSIVE" or "ACCESS EXCLUSIVE"... or two-phase commit or idempotency for the operations you're doing.

[0] https://www.postgresql.org/docs/current/explicit-locking.htm...

Re: How to do distributed locking (2016)

#84

Post by Quekid5 »

candiddevmike wrote: post #34
eknkc wrote: post #25

I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something. Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some…

One gotcha maybe with locks is they are connection specific AFAIK, and in most libraries you're using a pool typically. So you need to have a specific connection for locks, and ensure you're using that connection when doing periodic lock tests.

Why would locks be connection-specific? ... considering that only one operation can be in flight at a time on a single connection. (Usually, at least.)

Re: How to do distributed locking (2016)

#85

Post by joatmon-snoo »

Quekid5 wrote: post #84
candiddevmike wrote: post #34

Earlier quoted context omitted.

One gotcha maybe with locks is they are connection specific AFAIK, and in most libraries you're using a pool typically. So you need to have a specific connection for locks, and ensure you're using that connection when doing periodic lock tests.

Why would locks be connection-specific? ... considering that only one operation can be in flight at a time on a single connection. (Usually, at least.)

Different DBs implement locks differently.

Postgres allows obtaining advisory locks at either the session _or_ transaction level. If it's session-level, then you have, ergo, a connection-level lock.

https://www.postgresql.org/docs/current/explicit-locking.htm...

Re: How to do distributed locking (2016)

#86

Post by sethammons »

shepherdjerred wrote: post #65
noprocrasted wrote: post #26

Earlier quoted context omitted.

> there isn’t an alignment of incentives that fixes this problem "Microservices" itself is often a symptom of this problem. Everyone and their dog wants to introduce a network boundary in between function calls for no good reason just so they can subsequently have endless busywork writing HTTP (or gRPC if you're lucky) servers, clients & JSON (de?)serializers for said function calls and try to reimplement things like…

If you're hand-writing clients/servers/serializers instead of generating them from schema definitions then you have more fundamental issues than using microservices.

We hand wrote clients at my last microservice based gig. It was marginally slower than automated clients and we did run into a few cases of teams "waisting" their time writing their own clients; that was fixed by the authoring service team also authoring clients. It wasn't a big issue

Re: How to do distributed locking (2016)

#87

Post by jroseattle »

nh2 wrote: post #35
jroseattle wrote: post #27

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a reques…

You are right that anything that needs up to 50000 atomic, short-lived transactions per second can just use Postgres. Your UPDATE transaction lasts just a few microseconds, so you can just centralise the problem and that's good because it's simpler, faster and safer. But this is not a _distributed_ problem, as the article explains: > remember that a lock in a distributed system is not like a mutex in a multi-threaded…

> up to 50000 atomic, short-lived transactions per second

50000?

> You need distributed locking if the transactions can take seconds or hours, and the machines involved can fail while they hold the lock. From my experience, locks are needed to ensure synchronized access to resources. Distributed locks are a form of that isolation being held across computing processes, as opposed to the mutex example provided.

And while our implementation definitively did not use a distributed lock, we could still see those machines fail.

I fail to understand why a distributed lock is needed for anything due to it's duration.

Re: How to do distributed locking (2016)

#88

Post by jroseattle »

stickfigure wrote: post #38
jroseattle wrote: post #27

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a reques…

I think this illustrates something important, which is that: You don't need locking . You need . In your case, the constraint is "don't sell more than N tickets". For most realistic traffic volumes for that kind of problem, you can solve it with traditional rdbms transactional behavior and let it manage whatever locking it uses internally. I wish developers were a lot slower to reach for "I'll build distributed locks…

This is exactly how we arrived at our solution. We needed to satisfy the constraint; locking was one means of addressing the constraint.

Maybe we were lucky in our implementation, but a key factor for our decision was understanding how to manage the systems in our environment. We would have skilled up with Redis, but we felt our Postgres solution would be a good first step. We just haven't had a need to go to a second step yet.

Re: How to do distributed locking (2016)

#89

Post by winwang »

dataflow wrote: post #82
winwang wrote: post #80

Earlier quoted context omitted.

Everything you're saying is plausibly possible in the absurdly large search space of all possible scenarios. The author's premise, however, is rooted in the specific scenario they lay out, with historical supporting examples which you can look into. Even then, the premise before all that was essentially: Redlock does not do what people might expect of a distributed lock. Btw I do have responses to your questions, but…

> The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly". I get that -- and honestly, I'm not expecting a treatise on distributed consensus here. But what took me aback was that the blog post didn't even attempt to mention anything about the fact that the premise (at first glance) looks glaringly broken. If he'd even said 1 single sentence like "it's {difficult/infeasible/impossible}…

Yeah, that makes sense now. I think, personally, I've simply seen that design around a bunch, but great on you to question it and call it out -- also plausible that my own headcanon doesn't check out.

Re: How to do distributed locking (2016)

#90

Post by dataflow »

winwang wrote: post #89
dataflow wrote: post #82

Earlier quoted context omitted.

> The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly". I get that -- and honestly, I'm not expecting a treatise on distributed consensus here. But what took me aback was that the blog post didn't even attempt to mention anything about the fact that the premise (at first glance) looks glaringly broken. If he'd even said 1 single sentence like "it's {difficult/infeasible/impossible}…

Yeah, that makes sense now. I think, personally, I've simply seen that design around a bunch, but great on you to question it and call it out -- also plausible that my own headcanon doesn't check out.

Thanks, yeah. For what it's worth, partly what led me to even leave this comment is that when he wrote "the code above is broken", I stared at it, and for the life of me I couldn't see why it was broken. Because, of course, the code was lying: there was no mention of leases or timeouts. Having a "lease" suddenly pulled out of nowhere really felt like a fast one being pulled on me (and really unfairly so!), hence I decided I'd actually leave the comment and question what the basis for this hidden time bomb even was in the first place?! If the code had said leaseLock(filename, timeout), I think the bug would've been glaringly obvious, and far fewer people would've been surprised by looking at the code.

Also for what it's worth, I can guess what some of the answers might be. For example, it's possible you'd need very precise timing facilities that aren't always available, in order to be able to guarantee high throughput with correctness (like Google Spanner's). Or it might be that doing so requires a trade-off between availability and partition-tolerance that in some applications isn't justified. But I'm curious what the answer actually is, rather than just (semi-)random guesses as to what it could be.

Post reply on HN