How to do distributed locking (2016) • Hacker News Board

Re: How to do distributed locking (2016)

#31

Post by [deleted] » Sun, Oct 20, 2024, 2:40 PM UTC

jmull wrote: post #13

karmakaze wrote: post #10

Earlier quoted context omitted.

As mentioned in the article, a non-100%-correct lock can be used for efficiency purposes. So basically use an imperfect locking mechanism for efficiency and a reliable one for correctness.

> and a reliable one for correctness To be clear, my point is don't use distributed locking for correctness. There are much better options. Now, the atomicity I mention implies some kind of internal synchronization mechanism for multiple requests, which could be based on locks, but those would be real, non-distributed ones.

[deleted]

Re: How to do distributed locking (2016)

#32

Post by wwarner » Sun, Oct 20, 2024, 2:43 PM UTC

jroseattle wrote: post #27

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a reques…

This is the best way, and actually the only sensible way to approach the problem. I first read about it here https://code.flickr.net/2010/02/08/ticket-servers-distribute...

Re: How to do distributed locking (2016)

#33

Post by karmakaze » Sun, Oct 20, 2024, 2:44 PM UTC

jameshart wrote: post #30

karmakaze wrote: post #9

Earlier quoted context omitted.

This is known as 'optimistic locking'. But I wouldn't call it a distributed locking mechanism.

Optimistic locks are absolutely a distributed locking mechanism, in that they are for coordinating activity among distributed nodes - but they do require the storage node to have strong guarantees about serialization and atomicity of writes. That means it isn’t a distributed storage solution, but it is something you can build over the top of a distributed storage solution that has strong read after write guarantees.

I normally see it as a version column in a database where it being with the data makes it non-distributed.

I'm not even sure how it could be used for exclusive update to a resource elsewhere--all clients will think they 'have' the lock and change the resource, then find out they didn't when they update the lock. Or if they bump the lock first, another client could immediately 'have' the lock too.

Re: How to do distributed locking (2016)

#34

Post by candiddevmike » Sun, Oct 20, 2024, 2:59 PM UTC

eknkc wrote: post #25

I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something. Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some…

One gotcha maybe with locks is they are connection specific AFAIK, and in most libraries you're using a pool typically. So you need to have a specific connection for locks, and ensure you're using that connection when doing periodic lock tests.

Re: How to do distributed locking (2016)

#35

Post by nh2 » Sun, Oct 20, 2024, 3:12 PM UTC

jroseattle wrote: post #27

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a reques…

You are right that anything that needs up to 50000 atomic, short-lived transactions per second can just use Postgres.

Your UPDATE transaction lasts just a few microseconds, so you can just centralise the problem and that's good because it's simpler, faster and safer.

But this is not a _distributed_ problem, as the article explains:

> remember that a lock in a distributed system is not like a mutex in a multi-threaded application. It’s a more complicated beast, due to the problem that different nodes and the network can all fail independently in various ways

You need distributed locking if the transactions can take seconds or hours, and the machines involved can fail while they hold the lock.

Re: How to do distributed locking (2016)

#36

Post by hansvm » Sun, Oct 20, 2024, 3:19 PM UTC

wwarner wrote: post #32

jroseattle wrote: post #27

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a reques…

This is the best way, and actually the only sensible way to approach the problem. I first read about it here https://code.flickr.net/2010/02/08/ticket-servers-distribute...

> only sensible way

That's a bit strong. Like most of engineering, it depends. Postgres is a good solution if you only have maybe 100k QPS, the locks are logically (if not necessarily fully physically) partially independent, and they aren't held for long. Break any of those constraints, or add anything weird (inefficient postgres clients, high DB load, ...), and you start having to explore either removing those seeming constraints or using other solutions.

Re: How to do distributed locking (2016)

#37

Post by wwarner » Sun, Oct 20, 2024, 3:23 PM UTC

hansvm wrote: post #36

wwarner wrote: post #32

Earlier quoted context omitted.

This is the best way, and actually the only sensible way to approach the problem. I first read about it here https://code.flickr.net/2010/02/08/ticket-servers-distribute...

> only sensible way That's a bit strong. Like most of engineering, it depends. Postgres is a good solution if you only have maybe 100k QPS, the locks are logically (if not necessarily fully physically) partially independent, and they aren't held for long. Break any of those constraints, or add anything weird (inefficient postgres clients, high DB load, ...), and you start having to explore either removing those seemi…

Ok fair; I'm not really talking about postgres (the link i shared uses mysql). I'm saying that creating a ticket server that just issues and persists unique tokens, is a way to provide coordination between loosely coupled applications.

Re: How to do distributed locking (2016)

#38

Post by stickfigure » Sun, Oct 20, 2024, 3:32 PM UTC

jroseattle wrote: post #27

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a reques…

I think this illustrates something important, which is that: You don't need locking. You need .

In your case, the constraint is "don't sell more than N tickets". For most realistic traffic volumes for that kind of problem, you can solve it with traditional rdbms transactional behavior and let it manage whatever locking it uses internally.

I wish developers were a lot slower to reach for "I'll build distributed locks". There's almost always a better answer, but it's specific to each application.

Re: How to do distributed locking (2016)

#39

Post by hoppp » Sun, Oct 20, 2024, 3:32 PM UTC

I did distributed locking with Deno, and Deno KV hosted by Deno Deploy.

Its using foundationdb, a distributed db. The deno instances running on local devices all connect to the same Deno KV to acquire the lock.

But using postgres, a select for update also works, the database is not distributed tho.

Re: How to do distributed locking (2016)

#40

Post by zbobet2012 » Sun, Oct 20, 2024, 3:45 PM UTC

wwarner wrote: post #37

hansvm wrote: post #36

Earlier quoted context omitted.

> only sensible way That's a bit strong. Like most of engineering, it depends. Postgres is a good solution if you only have maybe 100k QPS, the locks are logically (if not necessarily fully physically) partially independent, and they aren't held for long. Break any of those constraints, or add anything weird (inefficient postgres clients, high DB load, ...), and you start having to explore either removing those seemi…

Ok fair; I'm not really talking about postgres (the link i shared uses mysql). I'm saying that creating a ticket server that just issues and persists unique tokens, is a way to provide coordination between loosely coupled applications.

Yeah that's cookies. They are great.