Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Race Patterns in Go

eng.uber.com

Post reply on HN
Print view View original

101–110 of 205 posts

Re: Data Race Patterns in Go

#101

Post by morelisp »

masklinn wrote: post #19
morelisp wrote: post #18

Earlier quoted context omitted.

> Because if it could realloc internally then I don't think this trick is safe. Slices are 3 word values of (ptr, len, cap). They cannot be "realloced internally", changing any of those three things requires creating a new slice.

Of course but the new slice could be (ptr, len+1, cap+x) because realloc() was able to expand the buffer in-place. Which yields essentially the same behaviour as an append call with leftover capacity. But I guess realloc is a libc function, and Go probably goes for mmap directly and would implement its own allocator, and so might not do that. Unless / until they decide to add support for it.

If you start worrying about "what if underlying the Go runtime happens?" you'll find a lot worse than realloc. Luckily, in the absence of runtime bugs, you don't have to think about it.

Re: Data Race Patterns in Go

#102

Post by masklinn »

morelisp wrote: post #101
masklinn wrote: post #19

Earlier quoted context omitted.

Of course but the new slice could be (ptr, len+1, cap+x) because realloc() was able to expand the buffer in-place. Which yields essentially the same behaviour as an append call with leftover capacity. But I guess realloc is a libc function, and Go probably goes for mmap directly and would implement its own allocator, and so might not do that. Unless / until they decide to add support for it.

If you start worrying about "what if underlying the Go runtime happens?" you'll find a lot worse than realloc. Luckily, in the absence of runtime bugs, you don't have to think about it.

It's not a concern about C, it's a concern about the underlying possibility expressed in C terms: sizeclass arenas are useful to an allocator, that means slack, which means the opportunity for in-place allocation resizing.

You advocated the use of a very specific behaviour of `append` as a DID and possibly a correctness requirement of programs.

My worry is about whether this behaviour is a hard specification of the Go language, or just an implementation detail of the primary Go implementation. And how programs applying your recommendation would handle such behaviour changing.

Re: Data Race Patterns in Go

#104

Post by masklinn »

Santosh83 wrote: post #69
ohazi wrote: post #68

A dig against Rust I sometimes hear is "Oh, data race freedom isn't such a big deal, if you really need it, a garbage collected language like Java will give you that guarantee." So now I'm hearing that Go, a garbage collected language, doesn't guarantee data race freedom? I guess it's garbage collected but not "managed" by a runtime or something? Why go to all that effort to get off of C++ just to stop 30% short? The…

Memory safe, excellent tooling, excellent base std library, no manual memory management, static binaries, trivial cross compilation, trivial concurrent programming etc. These advantages are still not inconsiderable over C, although not C++. Yes, however, it is not as "safe" as Rust. But downside is C++ & Rust are harder to design for upfront.

> Memory safe [...] trivial concurrent programming

As this post demonstrates, data races are trivial and common in Go.

Because many of the core types (interface, slice, map) are non-thread-safe multiword structures, they also break memory safety: https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...

Re: Data Race Patterns in Go

#105

Post by masklinn »

Philip-J-Fry wrote: post #41
masklinn wrote: post #32

Earlier quoted context omitted.

I’m a bit doubtful as what you talk about is definitely a slice issue but it’s already an issue in completely sequential code if you reuse appended-to slices. So while it’s also an issue in concurrent code, it’s really no more so.

It is an issue in sequential code because as you say, that's just how slices work. But if you're always using the same variable you'll never encounter it because that slice can't change between you reading that variable and writing to it. Once concurrency is introduced you can now read from the same variable, but another goroutine may have written to the same slice in the meantime. That's why you must protect the rea…

> Once concurrency is introduced you can now read from the same variable, but another goroutine may have written to the same slice in the meantime. That's why you must protect the read and writes and synchronise them.

But, again, this can easily occur in sequential code as well: you call a function passing it the slice, it mutates the slice internally, it doesn't document that, or maybe the documentation is even wrong, you now hit this issue.

Re: Data Race Patterns in Go

#106

Post by christophilus »

hintymad wrote: post #47
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. I don't quite understand the hatred (to the point of shouting "using Java? Over my dead body), especially in startups, towards Java. I mean, it's a language, big deal. Java's ecosystem more than enough offsets whatever inefficiencies in the language itself, at least for building many of the internal CRUD services. B…

For me, Java and MySQL kind of died* when they became an Oracle thing. I just don’t want to go near anything that Oracle touches.

The other thing is that I tend to write little programs where simple deployment on a low-resource machine is desirable.

Go can handle that. Java kind of does the job with Graal now.

The JVM is incredible, though, and I love Clojure. I’m hoping that Loom + Graal helps to kickstart more competition in the “concurrent, parallel, simple to deploy” space.

* Died to me; obviously they’re both alive and well in the broad world.

Re: Data Race Patterns in Go

#107

Post by pkolaczk »

hintymad wrote: post #47
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. I don't quite understand the hatred (to the point of shouting "using Java? Over my dead body), especially in startups, towards Java. I mean, it's a language, big deal. Java's ecosystem more than enough offsets whatever inefficiencies in the language itself, at least for building many of the internal CRUD services. B…

> how to build low-latency applications with ease too

That's a bit of a stretch. Surely, you can build low-latency apps, but I'd be very careful with the "with ease" bit. Low-latency Java often means zero heap allocations, aggressive object avoidance / reuse, heavy use of primitive types everywhere, so it is very much low-level like C, only with no tools that even plain old C offers, e.g. no true stack-allocated structs and no pointers. And forget about all the high-level zero-cost abstractions that C++ and Rust offer.

Re: Data Race Patterns in Go

#108

Post by kaba0 »

Santosh83 wrote: post #69
ohazi wrote: post #68

A dig against Rust I sometimes hear is "Oh, data race freedom isn't such a big deal, if you really need it, a garbage collected language like Java will give you that guarantee." So now I'm hearing that Go, a garbage collected language, doesn't guarantee data race freedom? I guess it's garbage collected but not "managed" by a runtime or something? Why go to all that effort to get off of C++ just to stop 30% short? The…

Memory safe, excellent tooling, excellent base std library, no manual memory management, static binaries, trivial cross compilation, trivial concurrent programming etc. These advantages are still not inconsiderable over C, although not C++. Yes, however, it is not as "safe" as Rust. But downside is C++ & Rust are harder to design for upfront.

> trivial concurrent programming

If it is chock-full of race conditions, it is not trivial.

Re: Data Race Patterns in Go

#109

Post by kaba0 »

christophilus wrote: post #106
hintymad wrote: post #47

Earlier quoted context omitted.

> I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. I don't quite understand the hatred (to the point of shouting "using Java? Over my dead body), especially in startups, towards Java. I mean, it's a language, big deal. Java's ecosystem more than enough offsets whatever inefficiencies in the language itself, at least for building many of the internal CRUD services. B…

For me, Java and MySQL kind of died* when they became an Oracle thing. I just don’t want to go near anything that Oracle touches. The other thing is that I tend to write little programs where simple deployment on a low-resource machine is desirable. Go can handle that. Java kind of does the job with Graal now. The JVM is incredible, though, and I love Clojure. I’m hoping that Loom + Graal helps to kickstart more comp…

> For me, Java and MySQL kind of died* when they became an Oracle thing. I just don’t want to go near anything that Oracle touches.

Come on, that’s a cheap reason (for Java, for open source db I would also go with postgre but for different reasons). Java is one of the very few languages with a full specification (not “whatever our compiler does, that’s the spec”), it has plenty of fully independent full implementations that even pass one of the most detailed test suites for complete spec-compliance, and the platform is so so much ingrained in the biggest corporations that any one of the following companies could easily, single-handedly finance the future of Java if anything were to happen: Apple, Google, Microsoft, Amazon, Alibaba.

And for all the bad things one can without doubt throw at Oracle, they are surprisingly good at shepherding the language and platform. It has been growing in a very good direction with fast update cycle, it has state-of-the art research and development going on, and with Loom on the near horizon and Valhalla on the slightly further horizon I would say Java has one of the brightest futures ahead. Like, Valhalla would bring automagically a huge performance improvement for free, and Java is very competitive in performance as is.

Re: Data Race Patterns in Go

#110

Post by tialaramex »

oconnor663 wrote: post #91
sa46 wrote: post #71

Earlier quoted context omitted.

A data race and garbage collection are unrelated. A data race occurs when: > two or more threads in a single process access the same memory location concurrently, and at least one of the accesses is for writing. Rust provides compile time protection against data races with the borrow checker. Go provides good but imperfect runtime detection of data races with the race detector. Like most things in engineering, either…

They're unrelated in theory, but in practice a lot of garbage collected languages do try to turn data races into defined behavior. Java requires the JVM to implement some defined semantics for data races, though I think they're still considered terribly confusing in practice. Python prevents data races with the GIL, and JS prevents them by either not having threads at all or not letting them share memory. I think Go…

Java promises that any variables touched by a data race are still valid, and your program still runs but it offers no guarantees about what value those variables have, so the signed integer you're using to count stuff up from zero might be -16 now, which is astonishing, but your program definitely won't suddenly branch into a re-format disk routine for no reason as it would be allowed to do in C or C++

Go has different rules depending on whether you race a primitive (like int) or some data structure, such as a slice, which has moving parts inside. If you race a data structure you're screwed immediately, this is always Undefined Behaviour. But if you race a primitive, Go says the primitive's representation is now nonsense, and so you're fine if you don't look at it. If you do look at it, and all possible representations are valid (e.g. int in Go is just some bits, all possible bit values are ints, whereas bool not so much) you're still fine but Go makes no promises about what the value is, otherwise that's Undefined Behaviour again.

I don't think Go is really unique here. Java put a lot of work in to deliver the guarantees it has, and since they turned out to be inadequate to reason about programs which don't exhibit Sequential Consistency that was work wasted. Most languages which don't have the data race problem simply don't have concurrency which is, well it's not cheating but it makes them irrelevant. C has "Sequential Consistency" under this constraint too.

Post reply on HN