Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Race Patterns in Go

eng.uber.com

Post reply on HN
Print view View original

191–200 of 205 posts

Re: Data Race Patterns in Go

#191

Post by randomswede »

mmarq wrote: post #100
saghm wrote: post #55

Earlier quoted context omitted.

> Go does not have threads but something like "tasks". The fact that no thread handle is exposed allows for transparently moving these tasks across threads if the scheduler decides so. This doesn't stop there being "task handles" then, though? I think the point GP was making is that something that in most languages would be simple methods on a handle like "wait for this task to finish" or "stop this task" instead nee…

My (rather horrid) pattern to address this problem is to wrap the goroutine in a function that returns a channel receiver. When the goroutine ends it sends something to the channel and whatever called it can await the result or completion using the receiver.

I have, on occasion, used a similar pattern, but instead of sending something, I simply close the channel (usually with a "defer close(c)" at the beginning of the function/closure that encompasses the main code of the goroutine's work).

That way, if I end up having multiple waiters, they will all be able to proceed.

Re: Data Race Patterns in Go

#192

Post by masklinn »

Thaxll wrote: post #130
tialaramex wrote: post #124

Earlier quoted context omitted.

I'd say there's an excellent chance your assumptions about types you got from "popular libraries" is more conservative and that's why you never detected any issues. For example take the JSON decoder. If you have several tasks which can use some data from a JSON blob in parallel, is it OK if they all just share the same JSON decoder? If you're horrified because this seems obviously like a bad idea, that'll be why you…

Why would you share a decoder? It makes no sense since you need to decode just once.

You can decode multiple values from a stream.

In a language which focuses on concurrency correctness, the decoder would either be thread-safe (in which case you could use it as an input queue) or not be usable from multiple threads (in which case you’d clearly have to create the queue yourself).

Re: Data Race Patterns in Go

#193

Post by masklinn »

yencabulator wrote: post #144
tialaramex wrote: post #124

Earlier quoted context omitted.

I'd say there's an excellent chance your assumptions about types you got from "popular libraries" is more conservative and that's why you never detected any issues. For example take the JSON decoder. If you have several tasks which can use some data from a JSON blob in parallel, is it OK if they all just share the same JSON decoder? If you're horrified because this seems obviously like a bad idea, that'll be why you…

A json.Decoder holds on to a single io.Reader, using it concurrently to decode multiple things is just plain old absurd. How would that even work? https://pkg.go.dev/encoding/json#NewDecoder

> How would that even work?

The same way it works to do so sequentially?

Re: Data Race Patterns in Go

#194

Post by masklinn »

shp0ngle wrote: post #112
smasher164 wrote: post #48

This definitely matches my experience using Go at my previous organization. 1. Closures and concurrency really don't mix well. The loop variable capture in particular is very pernicious. There's an open issue to change this behavior in the language: https://github.com/golang/go/issues/20733 . 2. Yep. I've seen this problem in our codebase. I've grown to just be very deliberate with data that needs to be shared. Put i…

The first link notes that they don’t plan to change the semantics of the range loop, but they might dis-allow referencing it which is… uhh ok maybe? But go is really trying to keep backwards compatibility so they won’t just change this

Both changes break BC though. I see the underlying divergence of opinion as one of signalling: changing loop semantics would silently change code behaviour, even if it should usually be for the better there’s a bit of an 1172, whereas making the loop variable non-reference-able noisily breaks code which is almost guaranteed to be incorrect.

Re: Data Race Patterns in Go

#195

Post by tialaramex »

oconnor663 wrote: post #171
tialaramex wrote: post #169

Earlier quoted context omitted.

Well spotted. This is arguably a hole, albeit a deliberate one. In practice the main reason people do this is collecting some sort of metric, whose exact value is unimportant and which anyway isn't contemplated by the machine. If your program tries to actually act on this data then yeah, you have successfully made your own life unnecessarily exciting and debugging your program may be difficult. I think it's fair to s…

As an interesting example of doing something meaningful with relaxed arithmetic, Arc uses a relaxed fetch_add to increment the refcount: https://doc.rust-lang.org/src/alloc/sync.rs.html#1331-1343 . Decrementing, however, uses acquire-release. Apparently shared_ptr in C++ is similar.

Tried to follow this link on my phone and the browser blew up, so I'll look when I'm home. Doubtless in both cases (Rust and C++) people who are much smarter than me have reasoned that it's correct and perhaps if I read the link I'll agree. But if you just asked me off the cuff my guess would have been this wasn't safe and so they should be Acquire-Release.

Re: Data Race Patterns in Go

#196

Post by masklinn »

sharno wrote: post #163

Go picked the concurrency ideas of Erlang but then ignored the main safeguard that makes Erlang's concurrency fearless: Immutability. And if you feel that Erlang's lack of type safety is an issue, then Gleam has you covered.

> Go picked the concurrency ideas of Erlang but then ignored the main safeguard that makes Erlang's concurrency fearless: Immutability.

Not even immutability, isolation.

Though obviously immutability makes things less weird, the real gain in terms of concurrency is that you can’t touch any data other than your own (process’s), and for the most part erlang doesn’t “cheat” either: aside from binaries, terms are actually copied over when sent, each process having its own heap.

Sequential erlang could be a procedural language based around mutability and it wouldn’t much alter its reliability guarantees (assuming binaries remain immutable, or become COW).

Re: Data Race Patterns in Go

#197

Post by saghm »

Groxx wrote: post #152
saghm wrote: post #76

Earlier quoted context omitted.

Yeah, I generally think of the word "thread" as referring to OS threads and/or "green" threads depending on context (and in this case I thought it was clear what you were referring to!), but since the person who responded to you made the distinction, I figured I'd use their terminology when explaining what I thought you were saying.

I was just leveraging your already-top reply to reply to both of you, sorry about that :) I should've just done two comments. I think you and I are on the same page here. I think the main reason it doesn't exist is that go had no generics. It'd need to be another custom-generic type (Future[T] basically), and it would make it harder to pass around, just like channels. But since channels are generally intrusively-adde…

> I think the main reason it doesn't exist is that go had no generics. It'd need to be another custom-generic type (Future[T] basically), and it would make it harder to pass around, just like channels. But since channels are generally intrusively-added, they aren't part of the return signature, so they avoid that generic-return issue.

That's a good point I hadn't thought of! Naively I wan to say they could just "implicitly" make anything returned from a `go func` be passed to a channel and then have `go func` return a channel, but that would require doing a bit of type inference as well as deciding semantics for whether it's possible to get multiple values out of that return channel. It honestly seems like there are some interesting ideas here (e.g. having multiple yields out of a go routine that then get sent to an "output" channel, making a sort of generator-like thing, but I guess I'm not super surprised that Go didn't choose to go this route.

Re: Data Race Patterns in Go

#198

Post by Groxx »

eru wrote: post #184
Groxx wrote: post #161

Earlier quoted context omitted.

If someone is claiming the garbage collection means freedom from data races, they are unambiguously wrong. Garbage collectors solve double-free bugs and usually memory leaks due to cyclic references.

Yes. Though garbage collectors can have an indirect influence on the design of the language, that makes it easier to handle data races. (As an example, image how much simpler Rust would be, if they went with garbage collection. Or how much more machinery Haskell would need, if they went with Rust's memory management strategies.)

I'm not sure what Rust would gain from a garbage collector - it'd still need all of lifetimes for instance, because ownership is the necessary piece for preventing races.

Re: Data Race Patterns in Go

#200

Post by tialaramex »

kaba0 wrote: post #143
tialaramex wrote: post #110

Earlier quoted context omitted.

Java promises that any variables touched by a data race are still valid, and your program still runs but it offers no guarantees about what value those variables have, so the signed integer you're using to count stuff up from zero might be -16 now, which is astonishing, but your program definitely won't suddenly branch into a re-format disk routine for no reason as it would be allowed to do in C or C++ Go has differe…

> so the signed integer you're using to count stuff up from zero might be -16 now, which is astonishing Actually, if it is an int, it is guaranteed to not be any number not explicitly set to (java has no-out-of-thin-air guarantees for 32-bit primitives). In practice on every modern implementation it is true of 64-bit primitives as well. So the prototypical data race condition of incrementing a primitive counter from…

Ooh, I did not know this. Do you happen to know where the "no-out-of-thin-air" guarantee is for the 32-bit primitives? Presumably in the Memory model docs somewhere?
Post reply on HN