Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Race Patterns in Go

eng.uber.com

Post reply on HN
Print view View original

11–20 of 205 posts

Re: Data Race Patterns in Go

#11

Post by masklinn »

metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> The "Slices" example is just nasty!

I've got to say I'm not entirely clear on what they talk about specifically.

Is it simply that the `results` inside the goroutine will be desync'd from `myResults` (and so the call to myAppend will interact oddly with additional manipulations of results), or is it that the copy can be made mid-update, and `result` itself could be incoherent?

Re: Data Race Patterns in Go

#12

Post by morelisp »

metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

Without disagreeing that it's an enormous footgun, one good way to avoid such slice issues is to use the uncommon `a[x:y:z]` form to ensure the slice can't grow. As we're starting to write a lot of generic slice functions with 1.18, we're using this form in almost all of them which may add elements.

Re: Data Race Patterns in Go

#13

Post by morelisp »

bumper_crop wrote: post #3

Sorry to say, but these hit close to home for me. A lot of the synchronization paradigms in Go are easy to misuse, but lead the author into thinking it's okay. the WaitGroup one is particularly poignant for me, since the race detector doesn't catch it. I'll add one other data race goof: atomic.Value. Look at the implementation. Unlike pretty much every other language I've seen, atomic.Value isn't really atomic, since…

> atomic.Value isn't really atomic, since the concrete type can't ever change after being set.

How does this mean it's non-atomic? As far as I know you can still never Load() a partial Store(). (Also, even if it was possible, this would never be a good idea...)

Re: Data Race Patterns in Go

#14

Post by throwaway894345 »

metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> What exactly happens in these cases? How can I trust myself, as a fallible human being, to reason about such cases when I'm trying to efficiently roll up a list of results. :-/

For me: minimize shared mutable data. If I really can’t get rid of some shared mutable data, I mutex it or use atomics or similar. This works very well—I almost never run into data races this way, but it is a discipline rather than a technical control, so you might have to deal with coworkers who lack this particular discipline.

Re: Data Race Patterns in Go

#15

Post by masklinn »

morelisp wrote: post #12
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

Without disagreeing that it's an enormous footgun, one good way to avoid such slice issues is to use the uncommon `a[x:y:z]` form to ensure the slice can't grow. As we're starting to write a lot of generic slice functions with 1.18, we're using this form in almost all of them which may add elements.

> one good way to avoid such slice issues is to use the uncommon `a[x:y:z]` form to ensure the slice can't grow.

Do you mean you always use `a[x:y:y]` in order to ensure there is no extra capacity and any append will have to copy the slice?

Is append guaranteed to create a new slice (and copy over the data) if the parameter is at capacity? Because if it could realloc internally then I don't think this trick is safe.

Re: Data Race Patterns in Go

#16

Post by symfoniq »

masklinn wrote: post #11
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> The "Slices" example is just nasty! I've got to say I'm not entirely clear on what they talk about specifically. Is it simply that the `results` inside the goroutine will be desync'd from `myResults` (and so the call to myAppend will interact oddly with additional manipulations of results), or is it that the copy can be made mid-update, and `result` itself could be incoherent?

I have the same question.

They talk about the "meta fields" of a slice. Is the problem that these "meta fields" (e.g. slice length and capacity) are passed by value, and that by copying them, they can get out of sync between coroutines?

Re: Data Race Patterns in Go

#17

Post by metadat »

throwaway894345 wrote: post #14
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> What exactly happens in these cases? How can I trust myself, as a fallible human being, to reason about such cases when I'm trying to efficiently roll up a list of results. :-/ For me: minimize shared mutable data. If I really can’t get rid of some shared mutable data, I mutex it or use atomics or similar. This works very well —I almost never run into data races this way, but it is a discipline rather than a techni…

Absolutely, the disappointing part is that as code authors, we need to constantly remember about various (otherwise appealing and even encouraged by the language syntax and control constructs) footguns and "never approach such areas" of (totally valid) syntax.

Reminds me of programming in Javascript (it's extreme example, but the similarity is there).

Re: Data Race Patterns in Go

#18

Post by morelisp »

masklinn wrote: post #15
morelisp wrote: post #12

Earlier quoted context omitted.

Without disagreeing that it's an enormous footgun, one good way to avoid such slice issues is to use the uncommon `a[x:y:z]` form to ensure the slice can't grow. As we're starting to write a lot of generic slice functions with 1.18, we're using this form in almost all of them which may add elements.

> one good way to avoid such slice issues is to use the uncommon `a[x:y:z]` form to ensure the slice can't grow. Do you mean you always use `a[x:y:y]` in order to ensure there is no extra capacity and any append will have to copy the slice? Is append guaranteed to create a new slice (and copy over the data) if the parameter is at capacity? Because if it could realloc internally then I don't think this trick is safe.

> Because if it could realloc internally then I don't think this trick is safe.

Slices are 3 word values of (ptr, len, cap). They cannot be "realloced internally", changing any of those three things requires creating a new slice.

Re: Data Race Patterns in Go

#19

Post by masklinn »

morelisp wrote: post #18
masklinn wrote: post #15

Earlier quoted context omitted.

> one good way to avoid such slice issues is to use the uncommon `a[x:y:z]` form to ensure the slice can't grow. Do you mean you always use `a[x:y:y]` in order to ensure there is no extra capacity and any append will have to copy the slice? Is append guaranteed to create a new slice (and copy over the data) if the parameter is at capacity? Because if it could realloc internally then I don't think this trick is safe.

> Because if it could realloc internally then I don't think this trick is safe. Slices are 3 word values of (ptr, len, cap). They cannot be "realloced internally", changing any of those three things requires creating a new slice.

Of course but the new slice could be (ptr, len+1, cap+x) because realloc() was able to expand the buffer in-place. Which yields essentially the same behaviour as an append call with leftover capacity.

But I guess realloc is a libc function, and Go probably goes for mmap directly and would implement its own allocator, and so might not do that. Unless / until they decide to add support for it.

Re: Data Race Patterns in Go

#20

Post by aaronbee »

masklinn wrote: post #11
metadat wrote: post #8

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency" . Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as…

> The "Slices" example is just nasty! I've got to say I'm not entirely clear on what they talk about specifically. Is it simply that the `results` inside the goroutine will be desync'd from `myResults` (and so the call to myAppend will interact oddly with additional manipulations of results), or is it that the copy can be made mid-update, and `result` itself could be incoherent?

I believe they made a mistake with that example. It doesn't look unsafe to me because the myResults sliced passed to the goroutine is not used. Or perhaps the racy part was left out of their snippet.

Below is what might be what they have meant. This code snippet is racy because an unsafe read of myResults is done to pass it to the goroutine and then that version of myResults is passed to safeAppend:

  func ProcessAll(uuids []string) {
    var myResults []string
    var mutex sync.Mutex
    safeAppend := func(results []string, res string) {
      mutex.Lock()
      myResults = append(myResults, res)
      mutex.Unlock()
    }

    for _, uuid := range uuids {
      go func(id string, results []string) {
        res := Foo(id)
        safeAppend(myResults, id)
      }(uuid, myResults) #
EDIT: Formatting and clarity
Post reply on HN