Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What could Go wrong with a mutex

evilmartians.com

Post reply on HN
Print view View original

21–30 of 59 posts

Re: What could Go wrong with a mutex

#21

Post by masklinn »

eptcyka wrote: post #14
hnlmorg wrote: post #4

People talk about the lack of generics et al as being Go's downfall but personally what I've found to be the biggest annoyance is the very thing Go boasts to be good at: writing bug/race free multi-threaded code. Channels are vastly oversold given their actual suitability and worse still, add their own classes of race conditions (deadlocks) so you often fallback to using mutexes for those, hopefully rare, occasions y…

Generics could for instance have helped implement a mutex that disallows access to the memory that needs to be synchronized without locking it first. This of course wouldn't have helped here, as the root cause was holding on to the mutex for too long. I was under the impression that Go had first party tooling for deadlock checking, something called threadanalyzer if I recall correctly.

> Generics could for instance have helped implement a mutex that disallows access to the memory that needs to be synchronized without locking it first.

Did any language actually do that before rust?

It's also somewhat less useful without compiler-checked ownership as you can easily have the protected data escape the lock.

Re: What could Go wrong with a mutex

#22

Post by tialaramex »

paavohtl wrote: post #7
Ameo wrote: post #6

The goroutine-inspect tool they used looks really useful. Does anyone know if there's a similar tool for Rust? I've had a few times in the past where I'm trying to figure out why some async application using tokio is hanging and having a similar tool would be very helpful.

There is Loom[1] (part of the Tokio project) for exhaustively testing multithreaded code. Though as far as I can tell it is designed for debugging threads, not async tasks. [1] https://github.com/tokio-rs/loom

Indeed. Loom is about simulating what-happens-before questions for concurrency, particularly Atomics.

Loom would show you if your program, in which you believed A must happen, then B or C, and then either D or the other of B and C; actually could sometimes result in D happening, then B, then A because you screwed up.

Safe Rust promises that your program won't have Undefined Behaviour as a result of such a mistake, but its defined behaviour might still be extremely surprising to you. From Rust's point of view, "Open fuel valve, allow hydrogen gas to flood into engine room, then start fans, then try to light pilot flame" is no less "safe" than "Check fans are running, check pilot flame on, if so open fuel valve" but you probably don't agree. People can write safe Rust abstractions and this is popular in the embedded world, but ultimately program logic is somewhere a programmer's responsibility. Maybe you want to cause explosions in the engine room.

Re: What could Go wrong with a mutex

#23

Post by ndr »

masklinn wrote: post #16
ndr wrote: post #9

Earlier quoted context omitted.

Avoiding deadlocks is easy if you're lockfree (or otherwise non-blocking), the interesting bit is being data-race free: https://www.ponylang.io/discover/#what-makes-pony-different

> Avoiding deadlocks is easy if you're lockfree (or otherwise non-blocking) Only if you’re also concurrency free. If you can still have two actors waiting on one another, you’re letter-of-the-law deadlock-free, but your program is still in effect deadlocked. > the interesting bit is being data-race free: Data race freedom is much weaker (and easier) than deadlock freedom, or race condition freedom. Safe rust is free…

> Only if you’re also concurrency free. If you can still have two actors waiting on one another, you’re letter-of-the-law deadlock-free, but your program is still in effect deadlocked.

I'm not sure we're using the same definitions. If you're non-blocking you have system wide progress, I'm not sure your examples of two actors mutually waiting on each other qualify.

In any case I think Pony qualifies as wait-free too, which avoids this scenario. Of course one can still have an infinite for loop, or an infinite ping-pong between actors.

Re: What could Go wrong with a mutex

#24

Post by jonathanstrange »

hnlmorg wrote: post #4

People talk about the lack of generics et al as being Go's downfall but personally what I've found to be the biggest annoyance is the very thing Go boasts to be good at: writing bug/race free multi-threaded code. Channels are vastly oversold given their actual suitability and worse still, add their own classes of race conditions (deadlocks) so you often fallback to using mutexes for those, hopefully rare, occasions y…

I concur. Although I perfectly understand why they were chosen, Go's concurrent synchronization primitives are not a good design choice. The developers of Go wanted to have primitives out of which better abstractions can be built while leaving freedom to the programmer, it makes sense and is consistent with Go's philosophy. In my experience, however, there is more buggy concurrent code out there than code that works correctly, and this could have been avoided by some higher level abstraction like the actor model or Ada's tasking with Rendezvous (where the latter is not so different from channels, to be fair).

It's a real problem because of 3rd party libraries. Too many 3rd party libraries produce deadlocks in unusual scenarios, and it can take almost as much time to write the library on your own than to find out whether their use of goroutines is correct. Deadlocks are really everywhere. The same with sync.atomic by the way, the use of these primitives is rarely correct and people keep using them despite the warnings in the docs. I'm not claiming to be smarter, on the contrary, the problem really is that Go's choice of concurrency primitives is only suitable for concurrency experts and most programmers (including myself) aren't.

Re: What could Go wrong with a mutex

#25

Post by masklinn »

ot wrote: post #5

The write priority of RW locks is a classical pitfall. Even knowing about this, I ended up falling into the exact same bug described in the article at least a couple of times. If anyone is wondering why RW mutexes block new readers if there is a waiting writer, the reason is to prevent writer starvation: if, say, two threads continuously acquire and release read locks, there may never be a point in time in which no t…

Here this strikes me as an awful API though, it seems rather insane that `Schedule` would just block with no timeout or clear indication.

Re: What could Go wrong with a mutex

#26

Post by gpderetta »

masklinn wrote: post #21
eptcyka wrote: post #14

Earlier quoted context omitted.

Generics could for instance have helped implement a mutex that disallows access to the memory that needs to be synchronized without locking it first. This of course wouldn't have helped here, as the root cause was holding on to the mutex for too long. I was under the impression that Go had first party tooling for deadlock checking, something called threadanalyzer if I recall correctly.

> Generics could for instance have helped implement a mutex that disallows access to the memory that needs to be synchronized without locking it first. Did any language actually do that before rust? It's also somewhat less useful without compiler-checked ownership as you can easily have the protected data escape the lock.

sync-like classes [1] have been used for a long time in C++. I'm pretty sure that Stroustrup had one in his book. Of course it is always easy to subvert this kind of protection in C++ as you can easily leak references outside of the implicit critical section.

[1] https://gcc.godbolt.org/z/3Wefz3cjK

Re: What could Go wrong with a mutex

#27

Post by gnfargbl »

One way I often debug this kind of problem is to use go's built-in pprof tooling. You can insert a profiling server into your program in four lines: https://pkg.go.dev/net/http/pprof. Then, do something like

    go tool pprof 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1'
to get a goroutine profile of your running program, and use the web command to display it in a browser as a DAG. The stuck goroutine can usually be seen pretty easily on the graph.

Re: What could Go wrong with a mutex

#28

Post by gnfargbl »

klodolph wrote: post #3

I’ll say that using a worker pool is, IMO, a bit tricker than it sounds. The whole point of making goroutine creation cheap in the first place is that you can just spawn a ton of them, and then use blocking calls anywhere you want. If you have a pool of goroutines, then you open yourself to the condition that all the goroutines in the pool block, waiting for code to execute that can’t schedule because there are no av…

I don't think using a worker pool is so bad, and I usually use one in preference to spawning lots of goroutines. I tend to follow this pattern https://play.golang.org/p/KtCPYaDiVFl, in which synchronization between the workers and the master relies on the language's built-in features and does not use explicit mutexes.

> For example, if work blocks, do you give up the slot to other work? No easy answer!

That's true in languages where threads are expensive, but goroutines are so cheap that you can probably avoid the problem simply by running a very large pool of workers.

Re: What could Go wrong with a mutex

#29

Post by gpderetta »

ndr wrote: post #23
masklinn wrote: post #16

Earlier quoted context omitted.

> Avoiding deadlocks is easy if you're lockfree (or otherwise non-blocking) Only if you’re also concurrency free. If you can still have two actors waiting on one another, you’re letter-of-the-law deadlock-free, but your program is still in effect deadlocked. > the interesting bit is being data-race free: Data race freedom is much weaker (and easier) than deadlock freedom, or race condition freedom. Safe rust is free…

> Only if you’re also concurrency free. If you can still have two actors waiting on one another, you’re letter-of-the-law deadlock-free, but your program is still in effect deadlocked. I'm not sure we're using the same definitions. If you're non-blocking you have system wide progress, I'm not sure your examples of two actors mutually waiting on each other qualify. In any case I think Pony qualifies as wait-free too,…

> I'm not sure your examples of two actors mutually waiting on each other qualify

Of course it qualifies, that's a classical text book example of deadlock.

Re: What could Go wrong with a mutex

#30

Post by gpderetta »

oogali wrote: post #10

I would zoom in on the actual fix: shrinking the size of the critical section. When you use defer to release the lock, the entirety of your function becomes the critical section. If you add another bit of code that needs the same lock, your program goes boom. I've hit this same issue a few times when I write a new, naive HTTP handler function that needs concurrent map access, then throw apachebench at it before I com…

Exactly. Mutexes should be held for the shortest time possible (and you should avoid calling any function you don't know or the implementation details), yet RW mutexes only make sense if the critical section is large enough. Ergo RW mutexes very rarely make sense.
Post reply on HN