Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Race Patterns in Go

eng.uber.com

Post reply on HN
Print view View original

121–130 of 205 posts

Re: Data Race Patterns in Go

#121

Post by crabbygrabby »

shp0ngle wrote: post #89
jeremy_wiebe wrote: post #58

Earlier quoted context omitted.

That number really store out to me too. I’d be very curious how they decide what becomes a separate service. I’d also be curious if the 50m lines of code included generated code.

I did not work in Uber, but a similar company, and... it's a very political thing, usually. Taking an existing service and making it into 2 new microservices is a "thing you did". Suddenly, you have "impact" and can claim the new service as "yours". Everyone wants to be a king of their little kingdom.

You are me and I am you, I'll always be with you

Re: Data Race Patterns in Go

#122

Post by ryanschneider »

What’s up with the totally broken syntax highlighting in this post, at least on iOS? 2100 micro services and not one of them is a valid syntax highlighter for blog posts.

Edit: oh I see it highlights red and underlines every keyword. I find that incredibly distracting, so much so I assumed their highlighter was broken, but also just realized they are screenshots.

Re: Data Race Patterns in Go

#123

Post by sethammons »

erik_seaberg wrote: post #60
sidlls wrote: post #49

Earlier quoted context omitted.

I'm no fan of go, although I think it's better than many other languages for services, but the argument here is against the label "append", not the operation. It's a poor name for the operation, but the documentation is quite clear about what's going on. I'd argue that understanding the keywords and builtins of a language is the bare minimum an engineer should do before he starts writing anything in it.

s2 := append(s1, x) s3 := append(s1, y) shouldn’t be allowed, because what it’s likely to do is not what anyone meant. In a pass-by-value language, passing a slice or map by value should copy it, append should be a method that returns void, and passing a pointer should be the way to share state and avoid copies.

What would you expect that code to do?

I'd expect to have two different slices, s2 and s3, to contain all the same elements aside from the last.

[a, b, c, x] and [a, b, c, y] and s1 remains [a, b, c]

Re: Data Race Patterns in Go

#124

Post by tialaramex »

Thaxll wrote: post #114
Groxx wrote: post #27

Earlier quoted context omitted.

The lack of generics has forced all Go concurrency to be intrusive (i.e. implemented by the person using literally any concurrency ), and yeah. It's horrifyingly error-prone in my experience. It means everyone needs to be an expert, and lol, everyone is not an expert. Generics might save us from the simple, mechanical flaws. Expect to see `Locker ` and `Atomic ` types cropping up. And unbounded buffered thread-safe q…

"because there are an absurd amount of races in nearly all of the popular libraries" This is fud, I ran the race detector with a lot of popular lib and I never found issues like that. But since you're claiming there are issues everywhere, do you have examples?

I'd say there's an excellent chance your assumptions about types you got from "popular libraries" is more conservative and that's why you never detected any issues.

For example take the JSON decoder. If you have several tasks which can use some data from a JSON blob in parallel, is it OK if they all just share the same JSON decoder?

If you're horrified because this seems obviously like a bad idea, that'll be why you didn't find any trouble. In some other systems your programs would be needlessly slow and clunky as a result, but in Go your assumptions were appropriate.

It seems Groxx expects in this case that either the JSON decoder would work fine used this way, or, the documentation would highlight that you can't do this. Go chooses neither.

Here's Brad Fitzpatrick:

"The assumption when unstated is that things are not safe for concurrent use, that zero values are not usable, that implementations implement interfaces faithfully, and that only one return values is non-zero and meaningful."

These are some pretty important assumptions, or to look at it another way, potential foot guns.

Re: Data Race Patterns in Go

#125

Post by tialaramex »

jonathanstrange wrote: post #118
tialaramex wrote: post #111

Earlier quoted context omitted.

Data Race Freedom is, unsurprisingly, Freedom from Data Races. A Data Race is any time when there's concurrent modification of a memory value, on modern hardware with multiple simultaneous execution contexts those modifications could in some sense happen at the same moment. [NB: Data Races are a subset of Race Conditions. Race Conditions are sometimes just a fact about the world and you need to write programs that co…

AFAIK, you still have to be very careful since data races based on data dependencies can never be excluded in general, that is theoretically not possible. What you get is a guarantee that your program is not in an undefined state. There are still plenty of ways to shoot yourself in the foot with incorrect synchronization.

> AFAIK, you still have to be very careful since data races based on data dependencies can never be excluded in general

Hmm. Maybe I don't understand what you're getting at here. It seems like you're suggesting something like a[b] = x could race in safe Rust because we don't know b in advance and maybe it ends up being the same in two threads ?

But Rust's borrow checker won't allow both threads to have the same mutable array a so this is ruled out. You're going to have to either give them immutable references to a, which then can't be modified and so there's no data race, or else they need different arrays.

This is boringly easy to get right in theory, Rust just has to do a lot of work to make it usable while still delivering excellent runtime performance.

Re: Data Race Patterns in Go

#126

Post by throwaway894345 »

josefx wrote: post #117
Groxx wrote: post #27

Earlier quoted context omitted.

The lack of generics has forced all Go concurrency to be intrusive (i.e. implemented by the person using literally any concurrency ), and yeah. It's horrifyingly error-prone in my experience. It means everyone needs to be an expert, and lol, everyone is not an expert. Generics might save us from the simple, mechanical flaws. Expect to see `Locker ` and `Atomic ` types cropping up. And unbounded buffered thread-safe q…

> I also really wonder where all these "go makes concurrency a first-class concept" claims come from, Given that some of the main architects behind Go had K&R C as background I wouldn't be surprised if "first-class" just meant that the language defines both a memory model and primitives for threading. C had neither until it basically adopted both from C++11.

That’s certainly how I interpreted it. It never occurred to me to think it meant “thread handles” specifically.

Re: Data Race Patterns in Go

#127

Post by armitron »

likeabbas wrote: post #39
setr wrote: post #31

Earlier quoted context omitted.

For free? Of course not. At cost? … depends on what you’re charging me, and how much I’m getting

Good point. Right now I kind of see modern programming as two fold: 1) Loosely typed to get you what you want faster, but with some mistakes, and 2) Strongly typed that forces you to try harder, but ultimately better I'm usually happier with the latter. I find I become far more frustrated when I try to write python than I do something like Rust just because I know when I write Python that I will have mistakes I'll ha…

Except, Python is strongly typed.

Re: Data Race Patterns in Go

#128

Post by armitron »

jchw wrote: post #65
likeabbas wrote: post #39

Earlier quoted context omitted.

Good point. Right now I kind of see modern programming as two fold: 1) Loosely typed to get you what you want faster, but with some mistakes, and 2) Strongly typed that forces you to try harder, but ultimately better I'm usually happier with the latter. I find I become far more frustrated when I try to write python than I do something like Rust just because I know when I write Python that I will have mistakes I'll ha…

I greatly prefer Rust to Python, but slightly prefer Go to Rust in many situations. Python even with MyPy just feels sloppy, whereas Go does offer decently strong typing at least. As for Go vs Rust, the answer is in the ecosystem, and I blame that on the difficulty of maintaining high quality Rust crates. I know not everyone agrees with this explanation, but I think Rust being complicated makes it hard to offer zero…

There are no gradients of strong typing. Either a language is strongly typed or not. Python is strongly typed.

For someone with such strong opinions, you seem to lack certain fundamental knowledge which I suggest you rectify ASAP as it will definitely improve your programming skill.

Re: Data Race Patterns in Go

#129

Post by armitron »

Klasiaster wrote: post #34

My favorite example is the IP address type which is an alias for a slice of bytes (type IP []byte). Thus, it gets passed by reference instead of by value and you easily end up working on the same data even if you didn't plan to. This will just be a logical bug but there are data structures in Go which result in memory corruption and introduce the risk of (remote) code execution vulnerabilities.

> Thus, it gets passed by reference instead of by value

Everything in Go is passed by value, including slices.

Go has no reference types, but a lot of people think it does, and that’s a problem. An example of the much bigger problem of low barrier to entry programming, where lots of folks write code but have no deep understanding of the tools they use.

If there’s something that Go proves, it’s that one can’t make a language “idiot proof”. Rust has the same problem in terms of folks creating mess after mess with it, except it’s higher barrier to entry and gets a better caliber of people using it.

Re: Data Race Patterns in Go

#130

Post by Thaxll »

tialaramex wrote: post #124
Thaxll wrote: post #114

Earlier quoted context omitted.

"because there are an absurd amount of races in nearly all of the popular libraries" This is fud, I ran the race detector with a lot of popular lib and I never found issues like that. But since you're claiming there are issues everywhere, do you have examples?

I'd say there's an excellent chance your assumptions about types you got from "popular libraries" is more conservative and that's why you never detected any issues. For example take the JSON decoder. If you have several tasks which can use some data from a JSON blob in parallel, is it OK if they all just share the same JSON decoder? If you're horrified because this seems obviously like a bad idea, that'll be why you…

Why would you share a decoder? It makes no sense since you need to decode just once.
Post reply on HN