Go’s race detector has a mutex blind spot
doublefree.dev
Go’s race detector has a mutex blind spot
1–10 of 60 posts
Re: Go’s race detector has a mutex blind spot
#2You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing.
The right way to use the go race detector is:
1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod.
2. Have a nightly job that runs unit and integ tests, built with -race, and without caching, and if any races show up there, save the trace and hunt for them. It only works probabilistically for almost all significant real-world code, so you have to keep running it periodically.
3. Accept that you'll have, for any decently sized go project, a chunk of mysterious data-races. The upstream go project has em, most of google's go code has em, you will to. Run your code under a process manager to restart it when it crashes. If your code runs on user's devices, gaslight your users into thinking their ram or processor might be faulty so you don't have to debug races.
4. Rewrite your code in rust, and get something better than the go race detector every time you compile.
The most important of those is 3. If you don't do anything else, do 3 (i.e. run your go code under systemd or k8s with 'restart=always').
Re: Go’s race detector has a mutex blind spot
#3You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
The data race patterns in Go article from Uber is always a scary read.
Re: Go’s race detector has a mutex blind spot
#4You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
> Rewrite your code in rust, and get something better than the go race detector every time you compile.
Congrats, rustc forced you to wrap all your types in Arc>, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch.
Don’t get me wrong, Rust has done a ton for safety and pushed other languages to do better. I love probably 50% of Rust. But Rust doesn’t protect against logical races, lovelocks, deadlocks, and so on.
To write concurrent programs that have the same standards of testable, composable, expressive etc as we are expecting with sequential programs is really really difficult. Either we need new languages, frameworks or (best case) design- and architectural patterns that are easy to apply. As far as I’m concerned large scale general purpose concurrent software development is an unsolved problem.
Re: Go’s race detector has a mutex blind spot
#5You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
I configure ci to run tests with -race and that works out pretty well. I value short ci runs, so testing with -race is a sacrifice for me even if it only adds ~10 seconds typically. I like your idea of a regular job that runs without caching, but your best tip is gaslighting users. Maybe I should start prefixing error messages with “look what you made me do”.
Re: Go’s race detector has a mutex blind spot
#6You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…
If it's solved the solution has been discarded at some point by other developers for being too cumbersome, too much effort, and therefore in violation of some sacred principle of their job needing to be effortless.
Re: Go’s race detector has a mutex blind spot
#7You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…
A well formed Go program would have the same logical race conditions to manage as well.
The Arc is only needed when you truly need to mutably share data.
Rust like Go has the full suite of different channels and what other patterns to share data.
Re: Go’s race detector has a mutex blind spot
#8You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…
> Congrats, rustc forced you to wrap all your types in Arc>
Also, don’t people know that a Mutex implies lower throughput depending on how long said Mutex is held?
Lock-free data structures/algorithms are attempt to address the drawbacks of Mutexes.
https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadv...
Re: Go’s race detector has a mutex blind spot
#9You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…
I may be biased, as I definitely love more than 50% of Rust, but Go also does not protect against logical races, deadlocks, etc.
I have heard positive things about the loom crate[1] for detecting races in general, but I have not used it much myself.
But in general I agree, writing correct (and readable) concurrent and/or parallel programs is hard. No language has "solved" the problem completely.
Re: Go’s race detector has a mutex blind spot
#10You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing. The right way to use the go race detector is: 1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod. 2. Have a nightly job that runs unit and int…
> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc >, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for saf…
As a sibling said, Go has all the same deadlocks, livelocks, etc you point out that rust doesn't cover, in addition to also having data-races that rust would prevent.
But, also, Go has way worse semantics around various things, like mutexes, making it much more likely deadlocks happen. Like in go, you see all sorts of "mu.Lock(); f(); mu.Unlock()" type code, where if it's called inside an `http.Handler` and 'f' panics, the program's deadlocked forever. In go, panics are the expected way for an http middleware to abort the server ("panic(http.ErrAbortHandler)"). In rust, panics are expected to actually be fatal.
Rust's mutexes also gate "ownership" of the inner object, which make a lot of trivial deadlocks compiler errors, while go makes it absolutely trivial to forget a "mu.Unlock" in a specific codepath and call 'Lock' twice in a case rust's ownership rules would have caught.
In practice, for similarly sized codebases and similarly experienced engineers, I see only a tiny fraction of deadlocks in concurrent rust code when compared to concurrent go code, so like regardless that it's an "unsolved problem", it's clear that in reality, there's something that's at least sorta working.