Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What could Go wrong with a mutex

evilmartians.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: What could Go wrong with a mutex

#51

Post by gpderetta »

ndr wrote: post #50
gpderetta wrote: post #48

Earlier quoted context omitted.

Let's say I'm writing a trivial proxy which simply forwards data unchanged between two sockets. This is my pseudocode: while true: x = read s1 write s2, x y = read s2 write s1, y This can deadlock. If for example s1 stops sending data, read(s1) blocks, so the proxy won't forward any data from s2 back to s1, potentially leading to a full pipeline deadlock. Now, the above can be expressed in purely non blocking fashion…

I think you might not be able to write that program, there simply is no way to block in Pony (modulo FFI). The scheduler wakes up actors that have messages (in their local/private queue) to process, all actors can do is to consume their queue and send (wait-free) messages to other actors if they want to. From your example "read X" would be reading from the actor's queue, so it would not be blocking. If the message qu…

The async_{read,write} variant never blocks nor has a while loop, it is just a four continuations recursively tail calling each other. Yet it deadlocks (as it is a purely mechanical transformation of the synchronous variant). It still might be possible that the precise pony scoping rules for closures prevent writing that variant, but I would be surprised.

Re: What could Go wrong with a mutex

#52

Post by kgeist »

hnlmorg wrote: post #39
_ph_ wrote: post #31

Earlier quoted context omitted.

There are several patterns possible which resolve this problem, your third example actually shows that, as it uses a function which only does the container access in a separate function which uses defer - this guarantees that no other code is executed while the lock is held. Alternatively, you could use an anonymous function inside your function which immediately gets called and performs the locking, lookup and defer…

That doesn't address the overhead of defer though (point 2) which can be a problem if your locks are in a well travelled path. One might argue that if you have a hot path that depends on mutexes then perhaps you need to restructure your code to avoid sharing data (and thus the need for a mutex). And that point would be valid, generally speaking. But there's always that one edge case where it is unavoidable (or even s…

In recent versions defer is basically zero overhead if no loops are involved. It's just a function call at every function exit. Basically what you'd do yourself manually.

Re: What could Go wrong with a mutex

#53

Post by ot »

gpderetta wrote: post #30
oogali wrote: post #10

I would zoom in on the actual fix: shrinking the size of the critical section. When you use defer to release the lock, the entirety of your function becomes the critical section. If you add another bit of code that needs the same lock, your program goes boom. I've hit this same issue a few times when I write a new, naive HTTP handler function that needs concurrent map access, then throw apachebench at it before I com…

Exactly. Mutexes should be held for the shortest time possible (and you should avoid calling any function you don't know or the implementation details), yet RW mutexes only make sense if the critical section is large enough. Ergo RW mutexes very rarely make sense.

> RW mutexes only make sense if the critical section is large enough. Ergo RW mutexes very rarely make sense.

I disagree :) Exclusive locks have unavoidable cache-line ping-ponging on the state, which can make a contended lock very expensive. In a write-infrequently read-frequently scenario, mutexes can do clever things to avoid that, for example have core-local reader counts.

Also there's a middle-ground between "pointer swap" and "so large that that it will block the writers enough to notice" critical sections, and often readers are in that sweet spot.

Re: What could Go wrong with a mutex

#54

Post by gpderetta »

ot wrote: post #53
gpderetta wrote: post #30

Earlier quoted context omitted.

Exactly. Mutexes should be held for the shortest time possible (and you should avoid calling any function you don't know or the implementation details), yet RW mutexes only make sense if the critical section is large enough. Ergo RW mutexes very rarely make sense.

> RW mutexes only make sense if the critical section is large enough. Ergo RW mutexes very rarely make sense. I disagree :) Exclusive locks have unavoidable cache-line ping-ponging on the state, which can make a contended lock very expensive. In a write-infrequently read-frequently scenario, mutexes can do clever things to avoid that, for example have core-local reader counts. Also there's a middle-ground between "po…

Well, yes you enter the realm of asymmetric synchronization (I like seqlocks myself), but I feel these are no longer general purpose synchronisation primitives but have often application specific trade-offs. Generic RWlocks are usually not great and do not come with these optimization

Also, yes my critical sections are often closer to the pointer swap case that might color my expectations.

Re: What could Go wrong with a mutex

#55

Post by pornel »

Ameo wrote: post #6

The goroutine-inspect tool they used looks really useful. Does anyone know if there's a similar tool for Rust? I've had a few times in the past where I'm trying to figure out why some async application using tokio is hanging and having a similar tool would be very helpful.

There's one in the works: https://github.com/tokio-rs/console

Re: What could Go wrong with a mutex

#56

Post by ot »

gpderetta wrote: post #54
ot wrote: post #53

Earlier quoted context omitted.

> RW mutexes only make sense if the critical section is large enough. Ergo RW mutexes very rarely make sense. I disagree :) Exclusive locks have unavoidable cache-line ping-ponging on the state, which can make a contended lock very expensive. In a write-infrequently read-frequently scenario, mutexes can do clever things to avoid that, for example have core-local reader counts. Also there's a middle-ground between "po…

Well, yes you enter the realm of asymmetric synchronization (I like seqlocks myself), but I feel these are no longer general purpose synchronisation primitives but have often application specific trade-offs. Generic RWlocks are usually not great and do not come with these optimization Also, yes my critical sections are often closer to the pointer swap case that might color my expectations.

My expectations may be colored by having a pretty good default implementation of a shared mutex in our codebase :) (folly::SharedMutex)

I've had to resort to seqlocks + async reclamation (either RCU or hazard pointers, depending on the case) only in a handful of very very specific cases.

Re: What could Go wrong with a mutex

#57

Post by gpderetta »

ot wrote: post #56
gpderetta wrote: post #54

Earlier quoted context omitted.

Well, yes you enter the realm of asymmetric synchronization (I like seqlocks myself), but I feel these are no longer general purpose synchronisation primitives but have often application specific trade-offs. Generic RWlocks are usually not great and do not come with these optimization Also, yes my critical sections are often closer to the pointer swap case that might color my expectations.

My expectations may be colored by having a pretty good default implementation of a shared mutex in our codebase :) (folly::SharedMutex) I've had to resort to seqlocks + async reclamation (either RCU or hazard pointers, depending on the case) only in a handful of very very specific cases.

Thanks, I'll take a look at folly::SharedMutex!

Re: What could Go wrong with a mutex

#58

Post by MaxBarraclough »

masklinn wrote: post #16
ndr wrote: post #9

Earlier quoted context omitted.

Avoiding deadlocks is easy if you're lockfree (or otherwise non-blocking), the interesting bit is being data-race free: https://www.ponylang.io/discover/#what-makes-pony-different

> Avoiding deadlocks is easy if you're lockfree (or otherwise non-blocking) Only if you’re also concurrency free. If you can still have two actors waiting on one another, you’re letter-of-the-law deadlock-free, but your program is still in effect deadlocked. > the interesting bit is being data-race free: Data race freedom is much weaker (and easier) than deadlock freedom, or race condition freedom. Safe rust is free…

> If you can still have two actors waiting on one another, you’re letter-of-the-law deadlock-free, but your program is still in effect deadlocked.

How is it letter-of-the-law deadlock-free?

Re: What could Go wrong with a mutex

#59

Post by treekogreen »

_ph_ wrote: post #31
hnlmorg wrote: post #15

Earlier quoted context omitted.

I think deferring unlocks should be more openly described as an anti-pattern. I do appreciate the convenience of defer generally speaking but it has two problems that specifically hinder effective use of mutexes: 1. it allows you to roll up additional code that doesn't need to be inside the mutex, thus keeping other threads waiting longer for an unlock 2. Benchmarks have shown[1][2] that using defer is actually more…

There are several patterns possible which resolve this problem, your third example actually shows that, as it uses a function which only does the container access in a separate function which uses defer - this guarantees that no other code is executed while the lock is held. Alternatively, you could use an anonymous function inside your function which immediately gets called and performs the locking, lookup and defer…

You could simply place `l.unlock()` underneath the function call to remove the overhead of the defer.
Post reply on HN