Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Race Patterns in Go

eng.uber.com

Post reply on HN
Print view View original

141–150 of 205 posts

Re: Data Race Patterns in Go

#141

Post by Groxx »

sethammons wrote: post #123
erik_seaberg wrote: post #60

Earlier quoted context omitted.

s2 := append(s1, x) s3 := append(s1, y) shouldn’t be allowed, because what it’s likely to do is not what anyone meant. In a pass-by-value language, passing a slice or map by value should copy it, append should be a method that returns void, and passing a pointer should be the way to share state and avoid copies.

What would you expect that code to do? I'd expect to have two different slices, s2 and s3, to contain all the same elements aside from the last. [a, b, c, x] and [a, b, c, y] and s1 remains [a, b, c]

When printing s1, s2, then s3:

It does exactly that, yes: https://go.dev/play/p/rs2FeK_QUjs

    [a b c]
    [a b c x]
    [a b c y]
But maybe it doesn't: https://go.dev/play/p/Na-eL0sOV9e

    [a b c]
    [a b c y]
So... maybe they share the same backing array? Lets try setting s2[0] to "z" after appending with the original code: https://go.dev/play/p/mAB-gUb0shB

    [a b c]
    [z b c x]
    [a b c y]
Apparently not. But also apparently yes? https://go.dev/play/p/k1ciGzyS2gc

    [z b c]
Let's try appending just one more item before redoing ^ that example, where they all shared the same data: https://go.dev/play/p/5JneXHMeUjx

    [a b c]
    [z b c x x2]
    [a b c y y2]
Notice that in all of these examples, I haven't explicitly declared a length or capacity. There's nothing "funny looking" or clearly intentionally allowing these different behaviors, it's just simple, very-common slice use.

.... so yeah. This is a source of a number of hard-to-track-down bugs.

Re: Data Race Patterns in Go

#142

Post by Groxx »

throwaway894345 wrote: post #126
josefx wrote: post #117

Earlier quoted context omitted.

> I also really wonder where all these "go makes concurrency a first-class concept" claims come from, Given that some of the main architects behind Go had K&R C as background I wouldn't be surprised if "first-class" just meant that the language defines both a memory model and primitives for threading. C had neither until it basically adopted both from C++11.

That’s certainly how I interpreted it. It never occurred to me to think it meant “thread handles” specifically.

I meant it more in that "first-class" tends to mean "this is a thing that is represented in the language / type system".

Go has first-class functions, because you can make a `var fn func() string` field/variable/argument/etc that holds a reference to a func that returns a string.

Go does not have first-class types, because you can't reference or store a type directly. You can use reflection to pass a reflected thing representing a type, but not the type itself. Generics muddies this somewhat, but I'll argue that falls under "generics", not "first-class types". In contrast, Java has both generics and first-class types, because you can pass `SomeClass` itself as an argument.

---

Node.js arguably has first-class concurrency. It has async/await: you do not have concurrency without those keywords. If they exist, you have potential concurrency. If they do not, you do not. (there may be exceptions here for true thread use, and JS runtimes vary, but you get the idea)

Rust has async/await now, and also has Send/Sync, which gives it a very strong claim to "first-class concurrency".

Go's concurrency constructs have no representation in the type system. They're totally invisible. Channels and select are mostly used with concurrency, but they do not define concurrency, and can be (and are) used synchronously as well.

`go` is a keyword, but I don't see how that's any different than `new Thread(fn)`... except that the Thread has a better claim to first-class-ness, because it returns a value that represents the concurrently-executing thread. If you have a thread reference, you know that concurrency exists. The reverse is not true though.

Re: Data Race Patterns in Go

#143

Post by kaba0 »

tialaramex wrote: post #110
oconnor663 wrote: post #91

Earlier quoted context omitted.

They're unrelated in theory, but in practice a lot of garbage collected languages do try to turn data races into defined behavior. Java requires the JVM to implement some defined semantics for data races, though I think they're still considered terribly confusing in practice. Python prevents data races with the GIL, and JS prevents them by either not having threads at all or not letting them share memory. I think Go…

Java promises that any variables touched by a data race are still valid, and your program still runs but it offers no guarantees about what value those variables have, so the signed integer you're using to count stuff up from zero might be -16 now, which is astonishing, but your program definitely won't suddenly branch into a re-format disk routine for no reason as it would be allowed to do in C or C++ Go has differe…

> so the signed integer you're using to count stuff up from zero might be -16 now, which is astonishing

Actually, if it is an int, it is guaranteed to not be any number not explicitly set to (java has no-out-of-thin-air guarantees for 32-bit primitives). In practice on every modern implementation it is true of 64-bit primitives as well.

So the prototypical data race condition of incrementing a primitive counter from n threads can loose counts, but will never have any value outside the 0..TRUE_COUNT range.

Re: Data Race Patterns in Go

#144

Post by yencabulator »

tialaramex wrote: post #124
Thaxll wrote: post #114

Earlier quoted context omitted.

"because there are an absurd amount of races in nearly all of the popular libraries" This is fud, I ran the race detector with a lot of popular lib and I never found issues like that. But since you're claiming there are issues everywhere, do you have examples?

I'd say there's an excellent chance your assumptions about types you got from "popular libraries" is more conservative and that's why you never detected any issues. For example take the JSON decoder. If you have several tasks which can use some data from a JSON blob in parallel, is it OK if they all just share the same JSON decoder? If you're horrified because this seems obviously like a bad idea, that'll be why you…

A json.Decoder holds on to a single io.Reader, using it concurrently to decode multiple things is just plain old absurd. How would that even work?

https://pkg.go.dev/encoding/json#NewDecoder

Re: Data Race Patterns in Go

#145

Post by yencabulator »

bumper_crop wrote: post #35
fastest963 wrote: post #29

Earlier quoted context omitted.

This is why all the examples call Store immediately with a zero value of the type.

https://go.dev/play/p/xolc9oPwA0C Interfaces don't have a zero type, which means that we can't have an atomic.Value which stores Shape. Atomic Value would be much easier to reason about if it had store semantics similar to a regular `var foo Shape = ...`. One of the other comment threads talked about generics helping this, so maybe there is hope.

Parent means

    var bestShape atomic.Value
    bestShape.Store((*Circle)(nil))

Re: Data Race Patterns in Go

#147

Post by yencabulator »

bumper_crop wrote: post #22
morelisp wrote: post #13

Earlier quoted context omitted.

> atomic.Value isn't really atomic, since the concrete type can't ever change after being set. How does this mean it's non-atomic ? As far as I know you can still never Load() a partial Store(). (Also, even if it was possible, this would never be a good idea...)

That's why I opened with "Look at the implementation". Go is unable to store the type and the pointer at the same time, so it warps what "atomic" means. Pretty much every other language has atomic mean "one of these will win, one will lose". Go says "one will win, one will panic and destroy the goroutine. In fact, it's even worse than that. If the Store() caller goes to sleep between setting the type and storing the…

> If the Store() caller goes to sleep between setting the type and storing the pointer, it causes every Goroutine that calls Load() to block.

Where does this go to sleep: https://cs.opensource.google/go/go/+/refs/tags/go1.18.3:src/...

It looks like a CAS busy loop with preemption disabled, to me.

Re: Data Race Patterns in Go

#148

Post by hintymad »

pkolaczk wrote: post #107
hintymad wrote: post #47

Earlier quoted context omitted.

> I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. I don't quite understand the hatred (to the point of shouting "using Java? Over my dead body), especially in startups, towards Java. I mean, it's a language, big deal. Java's ecosystem more than enough offsets whatever inefficiencies in the language itself, at least for building many of the internal CRUD services. B…

> how to build low-latency applications with ease too That's a bit of a stretch. Surely, you can build low-latency apps, but I'd be very careful with the "with ease" bit. Low-latency Java often means zero heap allocations, aggressive object avoidance / reuse, heavy use of primitive types everywhere, so it is very much low-level like C, only with no tools that even plain old C offers, e.g. no true stack-allocated stru…

Fair point. The "with ease" part has also to do with Java's ecosystem. For instance, Martin Thompson used to teach people how to write a single-producer-multi-consumer queue. In a matter of hours, people can achieve 100M+ reads and writes on a 2014 MacBook Pro (I understand that throughput is different from latency, but given the fixed number of CPUs in this case, the latency of such implementation is also phenomenal). Better yet, Java folks have libraries like JCTools, so they don't event have to spend that few hours to get even higher performance.

My litmus test is how fast one can implement functionalities of the data structures/algorithms in the book The Art of Multiprocessor Programming in production quality. It looks chic languages like Rust are not there yet.

Re: Data Race Patterns in Go

#150

Post by yencabulator »

Groxx wrote: post #44
kosherhurricane wrote: post #37

Earlier quoted context omitted.

>>> The reference to the slice was resized in the middle of an append operation from another async routine. > What exactly happens in these cases? Go's append looks like this: mySlice = append(mySlice, newItem) To me, this makes it very clear that 1) mySlice pointer can now point to someplace entirely different in memory, and 2) there maybe new allocation. I write both Java and Go. For personal projects, I always cho…

The append pattern also implies the opposite of reality, in that it also (usually!) mutates mySlice. Which is the source of one of the two(?) possible races in that piece of code.

Go's append is pretty much C's realloc, and behaves very much the same; the pointer you get back may or may not be the passed-in pointer.

Also,

> If the capacity of s is not large enough to fit the additional values, append allocates a new, sufficiently large underlying array that fits both the existing slice elements and the additional values. Otherwise, append re-uses the underlying array.

https://go.dev/ref/spec#Appending_and_copying_slices

Post reply on HN