Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The case of a leaky goroutine

brainbaking.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: The case of a leaky goroutine

#71

Post by pdimitar »

iforgotpassword wrote: post #44

I'm gonna be that guy. The old man yelling at cloud. I don't get all this high level crap. Coroutines, goroutines, fibers, async/await. It's supposed to make concurrency easy and safe. But I just fail to build a working mental model for it. I get the rough idea, but every time there's an await I wonder where execution might jump next. And then you read stuff like this, how these super high level comfortable languages…

> I prefer multi threaded programming

Then you had sheltered and privileged career which I am finding out applies to a lot of folks on HN, apparently. So hold on to your cushy job because if you leave it you might find out the world has moved on long ago.

Especially Rust's async/await are instrumental to services that literally saturate their network link and would go higher if the link allowed more bandwidth. Normal multi-threaded code is folding under pressure somewhere at the 50_000 requests / sec mark on an average small-ish VPS. In the meantime the async/await code is saturating a 10GbE link.

Both Golang and Rust didn't make the perfect design choices though -- that much is true, sadly. But they are a big improvement over the status quo.

I'll grant you that Golang replaced one class of problems with another -- something I dislike as well. Wish it was stricter but it's good enough for 95% of all projects everywhere, if we have to be brutally honest with ourselves.

Re: The case of a leaky goroutine

#72

Post by kbolino »

usrnm wrote: post #49
kbolino wrote: post #28

Earlier quoted context omitted.

Goroutines in a single process map onto a fixed number of threads. Even if you have goroutine leaks, you should not have thread leaks. Your program may deadlock or run out of memory, but it will not take the whole system down (at least, not in this way).

> Goroutines in a single process map onto a fixed number of threads Not necessarilly true if you're using cgo

I'd love to find a deep dive on how goroutines and cgo actually interact.

As I understand it, a cgo function call yields the caller goroutine and then runs the C code directly on the underlying thread, blocking it from use by goroutines. When the function returns, the thread is freed up, and goroutines including the caller can be scheduled again. I'm not sure if the caller is guaranteed to run next, or if other goroutines can crowd it out. I would imagine it probably does run first, if only to receive the return values from the C function and then release its thread affinity. This whole process is notable for introducing some overhead to cgo calls which can be significant if cgo is used frequently.

While you can create new threads in C and thus create thread leaks that way, I don't think any of those threads will be used by the goroutine scheduler, which sticks with the pool of threads it manages.

EDIT: reading the runtime docs, it seems that GOMAXPROCS is not as hard of a limit as I thought:

> The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit. This package's GOMAXPROCS function queries and changes the limit.

I think cgo calls count as "system calls on behalf of Go code" for this purpose. Thus if you have GOMAXPROCS=1 and more than one goroutine and you make a cgo call from one of them, the scheduler may create a new thread so the other goroutines can still run. You don't need cgo to do this though, syscalls (explicitly or through Go's stdlib) can exhibit the same behavior.

So I think it is possible to leak threads this way, but to do so you would need to spawn goroutines calling cgo faster than the C code can return.

Re: The case of a leaky goroutine

#73

Post by hedora »

duped wrote: post #27
hedora wrote: post #4

Earlier quoted context omitted.

My #1 complaint about Rust is that leaking a future is safe. It means the compiler can’t check for async coroutine leaks, and it breaks the borrow checker’s ability to say “nothing else has a reference to this any more”. Anyway, we’re using golang for some stuff at work, and holy crap, I forgot how terrible it was to work in high level languages that don’t statically check for correct synchronization. If C++-style co…

> My #1 complaint about Rust is that leaking a future is safe. Do you mean futures that aren't polled to completion, tasks that aren't joined, or literal memory leaks that happen to own futures?

The first and third thing. (They're basically equivalent.)

You can start polling then do std::mem::forget on the future. At that point, the borrow checker thinks the future no longer exists. So, it is unsound to pass a reference with a bounded lifetime into a future (which is why you need all the references to be 'static if you pass something into a spawn, or you need to spray Arc everywhere).

Re: The case of a leaky goroutine

#74

Post by hedora »

masklinn wrote: post #10
hedora wrote: post #4

Earlier quoted context omitted.

My #1 complaint about Rust is that leaking a future is safe. It means the compiler can’t check for async coroutine leaks, and it breaks the borrow checker’s ability to say “nothing else has a reference to this any more”. Anyway, we’re using golang for some stuff at work, and holy crap, I forgot how terrible it was to work in high level languages that don’t statically check for correct synchronization. If C++-style co…

> My #1 complaint about Rust is that leaking a future is safe. There’s no other way given the leakpocalypse decision. You’d need an entirely new leak-proof language to fix that, and that means you need alternatives for Rc and Arc (or a way to prevent them creating a cycle).

I'd gladly give up Rc and Arc if it meant the borrow checker treated spawn() like a normal function call.

All the code I write is async, so the borrow checker is effectively broken for me. (Wrapping everything in Arc creates weird false sharing at runtime, and I don't want to spend time debugging that class of performance nonsense.)

Re: The case of a leaky goroutine

#75

Post by hedora »

Thaxll wrote: post #41
kccqzy wrote: post #37

Earlier quoted context omitted.

A quote from your link: > programmers are strongly encouraged to use appropriate synchronization to avoid data races Any time you need to "encourage" programmers to do the right thing, you have already failed in your language design. And I think OP agrees with me here. OP says "static checking of correct synchronization" which is irresponsibly absent from Go.

So it's absent from every language but Rust?

I don't know of any other multithreaded systems programming languages that check for data race freedom at compile time.

Single threaded languages are usually data race free. I imagine some multithreaded, purely functional languages are too (everything is immutable, and therefore cannot be modified in race with reads). Of course, SQL running in strictly serializable mode is too.

Of those, the only one of those that's an appropriate choice for systems software development is Rust. The Core C++ Guidelines are a runner up in my opinion: They dictate a subset of C++ that is safer, with the goal of backporting the Rust memory safety properties to C++. Swift has also done a lot in this space.

Re: The case of a leaky goroutine

#76

Post by duped »

hedora wrote: post #73
duped wrote: post #27

Earlier quoted context omitted.

> My #1 complaint about Rust is that leaking a future is safe. Do you mean futures that aren't polled to completion, tasks that aren't joined, or literal memory leaks that happen to own futures?

The first and third thing. (They're basically equivalent.) You can start polling then do std::mem::forget on the future. At that point, the borrow checker thinks the future no longer exists. So, it is unsound to pass a reference with a bounded lifetime into a future (which is why you need all the references to be 'static if you pass something into a spawn, or you need to spray Arc everywhere).

They're very different, because a future may be dropped before it is polled to completion (aka, cancellation). If you leak it, then the drop handlers are not called, and this can affect program correctness.

It is not unsound for a future to own a reference (in fact it's super common - how else would asnc methods work?). If you leak the future, it can't be polled and it can't be dropped, so any references will never be dereferenced. But also that's a pretty contrived example.

Like you could call std::mem::forget on a future that owns a tokio::sync::MutexGuard and then you'd have some problems with deadlock... but that's not an async issue, it's always incorrect to leak an RAII guard (same as if you leaked std::sync::MutexGuard)

tokio::spawn has a 'static bound because it simplifies things, not because there's some fundamental limitation of futures owning references.

Re: The case of a leaky goroutine

#77

Post by shp0ngle »

latchkey wrote: post #34
shp0ngle wrote: post #9

Didn't Uber have some leaky goroutine detector? I vaguely remember seeing something like that, 5 years ago... Ah yeah it's here. https://github.com/uber-go/goleak

Uber also made something called fx, which is fantastic. You don't have to use it, but when you do, it helps ensure that you organize your code in a way that becomes very easily testable. It enforces a modular approach to composing together golang services. Being more easily testable helps prevent bugs, like these leaky goroutines.

I have seen fx used in production and it was an unholy mess. I never wish this upon anyone. It makes Go into Java.

Re: The case of a leaky goroutine

#78

Post by latchkey »

shp0ngle wrote: post #77
latchkey wrote: post #34

Earlier quoted context omitted.

Uber also made something called fx, which is fantastic. You don't have to use it, but when you do, it helps ensure that you organize your code in a way that becomes very easily testable. It enforces a modular approach to composing together golang services. Being more easily testable helps prevent bugs, like these leaky goroutines.

I have seen fx used in production and it was an unholy mess. I never wish this upon anyone. It makes Go into Java.

I guess there are things one could do to screw it up. But in your opinion, what exactly made it a mess?

Re: The case of a leaky goroutine

#79

Post by roguecoder »

Cthulhu_ wrote: post #8
jerf wrote: post #2

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g... There's a library for it: https://github.com/sourcegraph/conc But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new l…

> The concept was formulated in 2016 by Martin Sústrik (creator of ZeroMQ) with his C library libdill, with goroutines as a starting point. It's fairly new; the thing (and I think you address it too) is that the pattern did not exist yet when Go was introduced. Go is averse to adding more things to its standard library, or indeed changing its core fundamentals; I think it's better to have one well-defined way of doin…

> it's better to have one well-defined way of doing things in a language, instead of adding the mental overhead of deciding between one or the other

That was the hypothesis of Golang, for sure.

I think we've seen that it is true in specific contexts. It seems like it's been particularly valuable to massive teams with less-experienced developers coming from academic computer science backgrounds, frequent turnover and high-coordination projects.

I don't think that property has proven to be valuable more globally. Consistency for consistency's sake is particularly costly when the "consistent" solution has significant downsides in some contexts.

On small teams, having consistency result from actual alignment is incredibly valuable, and a sign of a high-performing team. In those contexts I haven't seen consistency itself, enforced by an outside group and especially without a way to work around it when the "consistent" approach has a reason it sucks for a particular application, be similarly valuable.

Re: The case of a leaky goroutine

#80

Post by roguecoder »

iforgotpassword wrote: post #44

I'm gonna be that guy. The old man yelling at cloud. I don't get all this high level crap. Coroutines, goroutines, fibers, async/await. It's supposed to make concurrency easy and safe. But I just fail to build a working mental model for it. I get the rough idea, but every time there's an await I wonder where execution might jump next. And then you read stuff like this, how these super high level comfortable languages…

Just because your position is unpopular doesn't make it wrong.

async/await is particularly damaging because it breaks the paradigm of Javascript. That no one can draw a picture explaining it is a huge problem, and it encourages people to write callback hell code by hiding how ugly it is. Unfortunately, whether the code is pretty or ugly the massive webs of nested callbacks are still a source of massive complexity and potential failures.

I constantly see tests in the wild that are passing even though the assertions fail because people don't understand the concurrency model they are using. And at least 60% of the time there was no reason for that concurrency to exist in the first place, except that it's what the code example looks like.

Post reply on HN