Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Race Patterns in Go

eng.uber.com

Post reply on HN
Print view View original

31–40 of 205 posts

Re: Data Race Patterns in Go

#31

Post by setr »

likeabbas wrote: post #28
jchw wrote: post #7

This is pretty cool. 50 million lines of code is quite a large corpus to work off of. I'm surprised by some of them. For example, go vet nominally catches misuses of mutexes, so it's surprising that even a few of those slipped through. I wonder if those situations are a bit more complicated than the example. Obviously, the ideal outcome is that static analysis can help eliminate as many issues as possible, by restric…

> I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered Does anyone not want robustness of their language to cover their mistakes?

For free? Of course not.

At cost? … depends on what you’re charging me, and how much I’m getting

Re: Data Race Patterns in Go

#32

Post by masklinn »

Philip-J-Fry wrote: post #23
masklinn wrote: post #11

Earlier quoted context omitted.

> The "Slices" example is just nasty! I've got to say I'm not entirely clear on what they talk about specifically. Is it simply that the `results` inside the goroutine will be desync'd from `myResults` (and so the call to myAppend will interact oddly with additional manipulations of results), or is it that the copy can be made mid-update, and `result` itself could be incoherent?

So a slice consists of a pointer to a backing array, a length and a capacity. If you don't use a pointer and pass this slice around you will copy it. This is problematic because even though you copy it, you're still pointing at the same backing array. Therefore, a backing array with data like [1,2,3,4,5] could be pointed at by 2 slice headers (slice metadata) looking like A: {len: 2, cap: 10} [1,2] B: {len:5, cap: 10…

I’m a bit doubtful as what you talk about is definitely a slice issue but it’s already an issue in completely sequential code if you reuse appended-to slices.

So while it’s also an issue in concurrent code, it’s really no more so.

Re: Data Race Patterns in Go

#33

Post by aaronbwebber »

At least some of these would be caught by running your tests with race detection on? I haven't read the whole article yet but as soon as I read the loop variable one I was pretty sure I have written code with that exact bug and had it caught by tests...

https://go.dev/doc/articles/race_detector

Edit: at the _end_ of the post, they mention that this is the second of two blog posts talking about this, and in the first post they explain that they caught these by deploying the default race detector and why they haven't been running it as part of CI (tl;dr it's slower and more resource-expensive and they had a large backlog).

https://eng.uber.com/dynamic-data-race-detection-in-go-code/

Re: Data Race Patterns in Go

#34

Post by Klasiaster »

My favorite example is the IP address type which is an alias for a slice of bytes (type IP []byte). Thus, it gets passed by reference instead of by value and you easily end up working on the same data even if you didn't plan to. This will just be a logical bug but there are data structures in Go which result in memory corruption and introduce the risk of (remote) code execution vulnerabilities.

Re: Data Race Patterns in Go

#35

Post by bumper_crop »

fastest963 wrote: post #29
bumper_crop wrote: post #22

Earlier quoted context omitted.

That's why I opened with "Look at the implementation". Go is unable to store the type and the pointer at the same time, so it warps what "atomic" means. Pretty much every other language has atomic mean "one of these will win, one will lose". Go says "one will win, one will panic and destroy the goroutine. In fact, it's even worse than that. If the Store() caller goes to sleep between setting the type and storing the…

This is why all the examples call Store immediately with a zero value of the type.

https://go.dev/play/p/xolc9oPwA0C

Interfaces don't have a zero type, which means that we can't have an atomic.Value which stores Shape. Atomic Value would be much easier to reason about if it had store semantics similar to a regular `var foo Shape = ...`. One of the other comment threads talked about generics helping this, so maybe there is hope.

Re: Data Race Patterns in Go

#37

Post by kosherhurricane »

metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

>>> The reference to the slice was resized in the middle of an append operation from another async routine.

> What exactly happens in these cases?

Go's append looks like this:

mySlice = append(mySlice, newItem)

To me, this makes it very clear that 1) mySlice pointer can now point to someplace entirely different in memory, and 2) there maybe new allocation.

I write both Java and Go. For personal projects, I always choose go.

Re: Data Race Patterns in Go

#38

Post by aaronbwebber »

dubswithus wrote: post #21

> We developed a system to detect data races at Uber using a dynamic data race detection technique. This system, over a period of six months, detected about 2,000 data races in our Go code base, of which our developers already fixed ~1,100 data races. This isn't open source, correct?

Yes it is, it's part of the standard go toolchain as described in the first blog post in the series: https://eng.uber.com/dynamic-data-race-detection-in-go-code/

Re: Data Race Patterns in Go

#39

Post by likeabbas »

setr wrote: post #31
likeabbas wrote: post #28

Earlier quoted context omitted.

> I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered Does anyone not want robustness of their language to cover their mistakes?

For free? Of course not. At cost? … depends on what you’re charging me, and how much I’m getting

Good point. Right now I kind of see modern programming as two fold:

1) Loosely typed to get you what you want faster, but with some mistakes, and 2) Strongly typed that forces you to try harder, but ultimately better

I'm usually happier with the latter. I find I become far more frustrated when I try to write python than I do something like Rust just because I know when I write Python that I will have mistakes I'll have to fix in prod, vs when I write Rust I won't have those mistakes (although it'll take me longer to get something to prod)

Re: Data Race Patterns in Go

#40

Post by [deleted] »

masklinn wrote: post #11
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> The "Slices" example is just nasty! I've got to say I'm not entirely clear on what they talk about specifically. Is it simply that the `results` inside the goroutine will be desync'd from `myResults` (and so the call to myAppend will interact oddly with additional manipulations of results), or is it that the copy can be made mid-update, and `result` itself could be incoherent?

[deleted]
Post reply on HN