How to do distributed locking (2016)

Post by throwawaythekey » Mon, Oct 21, 2024, 6:07 AM UTC

jroseattle wrote: post #87

nh2 wrote: post #35

Earlier quoted context omitted.

You are right that anything that needs up to 50000 atomic, short-lived transactions per second can just use Postgres. Your UPDATE transaction lasts just a few microseconds, so you can just centralise the problem and that's good because it's simpler, faster and safer. But this is not a _distributed_ problem, as the article explains: > remember that a lock in a distributed system is not like a mutex in a multi-threaded…

> up to 50000 atomic, short-lived transactions per second 50000? > You need distributed locking if the transactions can take seconds or hours, and the machines involved can fail while they hold the lock. From my experience, locks are needed to ensure synchronized access to resources. Distributed locks are a form of that isolation being held across computing processes, as opposed to the mutex example provided. And whi…

Post by skrause » Mon, Oct 21, 2024, 6:38 AM UTC

Quekid5 wrote: post #83

eknkc wrote: post #25

I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something. Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some…

Advisory locks have many pitfalls, see [0]. AFAIK the only correct way to do what you probably thought you were doing is "EXCLUSIVE" or "ACCESS EXCLUSIVE"... or two-phase commit or idempotency for the operations you're doing. [0] https://www.postgresql.org/docs/current/explicit-locking.htm...

Post by m11a » Mon, Oct 21, 2024, 9:02 AM UTC

skrause wrote: post #92

candiddevmike wrote: post #34

Earlier quoted context omitted.

One gotcha maybe with locks is they are connection specific AFAIK, and in most libraries you're using a pool typically. So you need to have a specific connection for locks, and ensure you're using that connection when doing periodic lock tests.

PostgreSQL has pg_advisory_xact_lock which releases the lock automatically when the transaction is over.

Post by refset » Mon, Oct 21, 2024, 9:50 AM UTC

wslh wrote: post #96

jojolatulipe wrote: post #60

At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.

I just discovered Temporal, and I have to say thank you! From what I've seen so far, it seems like the holy grail for workflows, offering very clear high-level task management over complex infrastructure. Is Temporal unique in this space, or are there other alternatives of similar caliber? Given that it was spun off from Uber and is used by top vendors, it sounds like it’s been thoroughly battle-tested.

Post by amievil » Thu, Oct 31, 2024, 7:35 PM UTC

jojolatulipe wrote: post #60

At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.