Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go’s race detector has a mutex blind spot

doublefree.dev

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Go’s race detector has a mutex blind spot

#11

Post by Xeoncross »

TheDong wrote: post #10
klabb3 wrote: post #4

Earlier quoted context omitted.

> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…

As a sibling said, Go has all the same deadlocks, livelocks, etc you point out that rust doesn't cover, in addition to also having data-races that rust would prevent. But, also, Go has way worse semantics around various things, like mutexes, making it much more likely deadlocks happen. Like in go, you see all sorts of "mu.Lock(); f(); mu.Unlock()" type code, where if it's called inside an `http.Handler` and 'f' panic…

> and 'f' panics, the program's deadlocked forever

I don't see `mu.Lock(); f(); mu.Unlock()` anywhere really.

`mu.Lock(); defer mu.Unlock(); f();` is how everyone does it to prevent that possibility.

Re: Go’s race detector has a mutex blind spot

#12

Post by franticgecko3 »

TheDong wrote: post #2

You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…

> Have a nightly job that runs unit and integ tests

Not enough IMHO.

We run all tests on developer machines and CI with -race. Always.

It's probabilistic, so every developer 'make test' and every 'git push' is coverage.

Re: Go’s race detector has a mutex blind spot

#13

Post by speed_spread »

mr90210 wrote: post #8
klabb3 wrote: post #4

Earlier quoted context omitted.

> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…

> Congrats, rustc forced you to wrap all your types in Arc > Also, don’t people know that a Mutex implies lower throughput depending on how long said Mutex is held? Lock-free data structures/algorithms are attempt to address the drawbacks of Mutexes. https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadv...

The overhead of Mutex for uncontended cases is negligible. If Mutex acquisition starts to measurably limit your production performance, you have options but will probably need to reconsider the use of shared mutable anyway.

Re: Go’s race detector has a mutex blind spot

#14

Post by aleksi »

TheDong wrote: post #2

You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…

> It's too slow to run in prod to be worth it

I disagree there. It is reasonable to run a few service instances with a race detector. I have a few services where _all_ instances are running with it just fine.

Re: Go’s race detector has a mutex blind spot

#15

Post by Xeoncross »

I'm so glad to be out of the dark ages of parallelism. Complaining about Go's race detector or exactly which types of logical races Rust can't prevent is such a breath of fresh air compared to all those other single-core languages we're paid to write with that had threading, async, or concurrency bolted-on as an afterthought.

I can only hope Go and Rust continue to improve until the next language generation comes along to surpass them. I honestly can't wait, things improved so much already.

Re: Go’s race detector has a mutex blind spot

#16

Post by [deleted] »

Xeoncross wrote: post #15

I'm so glad to be out of the dark ages of parallelism. Complaining about Go's race detector or exactly which types of logical races Rust can't prevent is such a breath of fresh air compared to all those other single-core languages we're paid to write with that had threading, async, or concurrency bolted-on as an afterthought. I can only hope Go and Rust continue to improve until the next language generation comes alo…

[deleted]

Re: Go’s race detector has a mutex blind spot

#17

Post by ViewTrick1002 »

Xeoncross wrote: post #11
TheDong wrote: post #10

Earlier quoted context omitted.

As a sibling said, Go has all the same deadlocks, livelocks, etc you point out that rust doesn't cover, in addition to also having data-races that rust would prevent. But, also, Go has way worse semantics around various things, like mutexes, making it much more likely deadlocks happen. Like in go, you see all sorts of "mu.Lock(); f(); mu.Unlock()" type code, where if it's called inside an `http.Handler` and 'f' panic…

> and 'f' panics, the program's deadlocked forever I don't see `mu.Lock(); f(); mu.Unlock()` anywhere really. `mu.Lock(); defer mu.Unlock(); f();` is how everyone does it to prevent that possibility.

Until you have to call a slow function after the mutex access leading to the lock being held long enough to cause problems.

Now you either refactor into multiple functions, while ensuring all copies of possibly shared data when passing function arguments are correctly guarded or ”manually” unlock when you don’t need the mutex access anymore.

Re: Go’s race detector has a mutex blind spot

#18

Post by valyala »

mr90210 wrote: post #8
klabb3 wrote: post #4

Earlier quoted context omitted.

> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…

> Congrats, rustc forced you to wrap all your types in Arc > Also, don’t people know that a Mutex implies lower throughput depending on how long said Mutex is held? Lock-free data structures/algorithms are attempt to address the drawbacks of Mutexes. https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadv...

Lock-free data structures and algorithms access shared memory via various atomic operations such as compare-and-swap and atomic arithmetic. The throughout of these operations do not scale with the number of CPU cores. Contrary, the throughput usually reduces with the growing number of CPU cores because they need more time for synchronizing local per-CPU caches with the main memory. So, lock-free data structures and algorithms do not scale on systems with big number of CPU cores. It is preferred to use "shared nothing" data structures and algorithms instead, where every CPU core processes its own portion of state, which isn't shared among other CPU cores. In this case the local state can be processed from local per-CPU caches at the speed which exceeds the main memory read/write bandwidth and has smaller access latency.

Re: Go’s race detector has a mutex blind spot

#19

Post by Jyaif »

I always run my Go code with `-race`, but I feel more comfortable writing C++ multithreaded code than Go thanks to the thread sanitizer annotations ( `__attribute__((guarded_by(guard)))` and others in the family).

The annotation also help me discover patterns, like when most of the functions of a class have the same annotations, maybe it means that all the functions of the class should have the same annotations.

I really wish an equivalent to those annotations came to Go.

Re: Go’s race detector has a mutex blind spot

#20

Post by jerf »

ViewTrick1002 wrote: post #17
Xeoncross wrote: post #11

Earlier quoted context omitted.

> and 'f' panics, the program's deadlocked forever I don't see `mu.Lock(); f(); mu.Unlock()` anywhere really. `mu.Lock(); defer mu.Unlock(); f();` is how everyone does it to prevent that possibility.

Until you have to call a slow function after the mutex access leading to the lock being held long enough to cause problems. Now you either refactor into multiple functions, while ensuring all copies of possibly shared data when passing function arguments are correctly guarded or ”manually” unlock when you don’t need the mutex access anymore.

OK, but you're not in "Go"-specific problems any more, that's just concurrency issues. There isn't any approach to concurrency that will rigorously prevent programmers from writing code that doesn't progress sufficiently, not even going to the extremes of Erlang or Haskell. Even when there are no locks qua locks to be seen in the system at all I've written code that starved the system for resources by doing things like trying to route too much stuff through one Erlang process.
Post reply on HN