Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are Go maps sensitive to data races?

dave.cheney.net

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Are Go maps sensitive to data races?

#41

Post by chrisseaton »

kasey_junk wrote: post #37
_ak wrote: post #32

Earlier quoted context omitted.

> The next thing on that list is a high performance concurrent queue. That's called a channel in Go.

A channel is a simple abstraction around a giant mutex. It does not scale throughput very well and has terrible latency performance to boot. Something like the disruptor ( https://lmax-exchange.github.io/disruptor/ ) is more along the lines of what I was thinking of. Without generics support you have to write it over and over again for each use case.

A go channels implemented using a mutex? I would be surprised if that was the case (but it might be). I would imagine such a fundamental construct built into the language would be lock-free.

Re: Are Go maps sensitive to data races?

#42

Post by tomp »

chrisseaton wrote: post #34
alkonaut wrote: post #13

Any data structure that isn't explicitly designed for concurrent access will not work when used concurrently. Since a concurrent map (aka. Dictionary aka. Hashtable) is a lot more complex and slow than a non concurrent one, it's very very unlikely that Go would ship with its standard map type being a concurrent one. It would be a huge waste. More likely there is a separate concurrent map type. It's exactly the same i…

> Since a concurrent map (aka. Dictionary aka. Hashtable) is a lot more complex and slow than a non concurrent one, it's very very unlikely that Go would ship with its standard map type being a concurrent one. It would be a huge waste. We should develop a system where maps are unsynchronised by default, but when you access them from a second thread they are then converted transparently by the runtime to a thread-safe…

I'm not sure that's possible. If the first thread isn't explicitly synchronizing the memory, it won't even detect when the map was changed (made thread-safe) by the second thread!

A half-solution could be using very lightweight synchronization (e.g. a MVar containing an unsynchronized map) that's later converted as you describe, but it would still incur some cynchronization overhead (even for single-threaded use).

Re: Are Go maps sensitive to data races?

#43

Post by kasey_junk »

chrisseaton wrote: post #41
kasey_junk wrote: post #37

Earlier quoted context omitted.

A channel is a simple abstraction around a giant mutex. It does not scale throughput very well and has terrible latency performance to boot. Something like the disruptor ( https://lmax-exchange.github.io/disruptor/ ) is more along the lines of what I was thinking of. Without generics support you have to write it over and over again for each use case.

A go channels implemented using a mutex? I would be surprised if that was the case (but it might be). I would imagine such a fundamental construct built into the language would be lock-free.

https://golang.org/src/runtime/chan.go

Lines 28, 152, 401

Re: Are Go maps sensitive to data races?

#44

Post by lambda_cube »

lazyjones wrote: post #31
alkonaut wrote: post #21

Earlier quoted context omitted.

I don't understand why someone would write a blog post about a collection type not being thread safe? Why not just look at the documentation and simply confirm that it isn't thread safe? Edit: Ok one reason could be that the documentation sucks and there are no mentions of the concurrency guarantees of the standard library types...

Possibly because the Go "authorities" are now trying to attract the kind of programmers who need "trigger warnings" and "safe spaces", for some reason. Just look at the new community CoC. I apologize if this constitutes a micro- or nano-aggression. More seriously, it's probably just a frequent mistake and they try to educate people proactively to help code quality in the long run. Edit: the non-atomic map updates are…

CoC? Code of Conduct? Churches of Christ? Call of Cthulhu?

Re: Are Go maps sensitive to data races?

#45

Post by chrisseaton »

kasey_junk wrote: post #43
chrisseaton wrote: post #41

Earlier quoted context omitted.

A go channels implemented using a mutex? I would be surprised if that was the case (but it might be). I would imagine such a fundamental construct built into the language would be lock-free.

https://golang.org/src/runtime/chan.go Lines 28, 152, 401

I stand corrected. I guess it must have some more complex functionality that I thought, needing a lock.

Re: Are Go maps sensitive to data races?

#46

Post by 4ad »

seanmcdirmid wrote: post #36
4ad wrote: post #4

Earlier quoted context omitted.

> Ahem Maps are not concurrency-safe, you know what else isn't? Integers, floats, pointers, structs. And not only in Go, but in pretty much almost all languages with mutable state, like C and C++. The map implementation in Go is consistent with the rest of the language, plus it allows for great performance under some rather common scenarios. But I guess it's fashionable to snark on message boards than rather understa…

32 bit data types are read and written atomically on the CLR. Pointers are read/written atomicly on most ANY platform, or you couldn't build any kind of higher level concurrency abstraction. If not, the language could never provide memory safety.

You need a way to do atomic reads and writes, but you don't need to have atomic types. In Go you just use sync/atomic which implements atomicity outside the type system.

When you use "regular" operations, you don't get atomicity, but what you get is well-defined, and explained here: https://golang.org/ref/mem.

Re: Are Go maps sensitive to data races?

#47

Post by grabcocque »

alkonaut wrote: post #16
coldtea wrote: post #7

Earlier quoted context omitted.

Yeah, but the parent's phrase was: "If you have a non-concurrency-safe map implementation in a language that's advertised as being good for concurrency, there is definitely something wrong". If we accept his premise, then the fact that "they wasn't designed for that" is no excuse. It's like selling a kids toy that has tiny choke-inducing parts. In that case, just saying: "It wasn't designed to be eaten by kids" is no…

A lot of comments that just seem to ignore the fact that noone would actually want standard collections to be thread safe! Concurrent collections are slow . That's why class libraries use different types for a therad safe collection and a regular non-thread safe collection. If you are using concurrency you have to use concurrent collections. You could argue that if most Go code is concurrent then the regular collecti…

They're really not though. Java has ConcurrentHashMap and isn't slow. Clojure has its persistent HashMap and it isn't slow. Scala has, TWO (?) concurrent hash maps in its core and neither is slow.

You know, two thirds of the way through the second decade of the 21st century, it's okay to spend a few cycles not corrupting your data structures.

Re: Are Go maps sensitive to data races?

#48

Post by chrisseaton »

tomp wrote: post #42
chrisseaton wrote: post #34

Earlier quoted context omitted.

> Since a concurrent map (aka. Dictionary aka. Hashtable) is a lot more complex and slow than a non concurrent one, it's very very unlikely that Go would ship with its standard map type being a concurrent one. It would be a huge waste. We should develop a system where maps are unsynchronised by default, but when you access them from a second thread they are then converted transparently by the runtime to a thread-safe…

I'm not sure that's possible. If the first thread isn't explicitly synchronizing the memory, it won't even detect when the map was changed (made thread-safe) by the second thread! A half-solution could be using very lightweight synchronization (e.g. a MVar containing an unsynchronized map) that's later converted as you describe, but it would still incur some cynchronization overhead (even for single-threaded use).

You could use the existing hardware memory protection mechanism, if you can do that per-thread. Each thread allocates into memory protected from other threads by default, and on a page fault caused by access from another thread it's then moved into shared memory, and converted to concurrent if needed.

You could then profile allocation sites so that if a map is frequently converted to concurrent, you then start allocating it as concurrent in the first place.

Re: Are Go maps sensitive to data races?

#50

Post by khgvljhkb »

Golang devs should have a look at Clojure and it's `core.async` library. It works very similarly to Go (heavily inspired by the good stuff), but all data structures are persistent, meaning you will never have problems with concurrent mutations (because there there are no mutations in the API, only internally).

I recommend Go users to install the boot utility, and playing around with Clojure & core.async. These communities can share many things.

Post reply on HN