Live data from Hacker News

The case of a leaky goroutine

brainbaking.com

31–40 of 87 posts

Re: The case of a leaky goroutine

#31
post #30

A lot of these problems come from accepting a line in the OP, "A Goroutine is essentially a coroutine". The rest of the sentence is "...that maps onto green threads that map onto real native threads on your OS in an NxM way". This is not a coroutine at all, calling them goroutines was a clever hacker pun along the lines of "GNU's Not Unix". If you treat a preëmptively-scheduled primitive as though it's a cooperativel…

I was under the impression that goroutines were cooperative but that they yield pretty aggressively (on blocking channel operations, disk and network I/O, cgo calls, and certain syscalls). What makes you think they're preemptive?

New implicit yield points started getting added from 1.2, which added a yield point in the function prologue (so any function call, even with no IO whatsoever, could yield). I think later releases also added yield points on allocation and stack growth.

This culminated in 1.14, which made the runtime preemptive on most platforms (https://go.dev/doc/go1.14#runtime) in order to fix the last sticking point where a goroutine might not yield: a tight numerical loop might never yield.

This was an issue, because the GC relied on scheduling to slip in its STW pauses, so the GC would trigger STW, progressively pause every goroutine reaching a yield point, but would be unable to ever pause the last goroutine, and the program would pretty much grind to a halt until it was done.

There are ways to handle this (e.g. insert trapping reads in various control structures), but ultimately preemption was considered a better and more useful solution.

Re: The case of a leaky goroutine

#32
post #24

Earlier quoted context omitted.

> just terminating a goroutine isn't safe with respect to I/O and defer chains. Agreed. The idea is to panic() if a goroutine has to be forcibly terminated due to GC, instead of a slow leak. Requires more thought though.

So the idea is to randomly take down the program when the GC runs?

Let me try to explain. The current way (which must continue working the same way even if the language is changed) does not allow you to distinguish a goroutine that should terminate but hasn't due to some bug versus a goroutine that can legitimately run for a long time. Making the goroutine "id" explicit can allow you distinguish the two cases. Store the id in some global or long lived variable or array if you want the goroutine to run for a long time. Otherwise carefully control the scope of such an id so that the runtime has a chance to catch the first case. That is my initial thinking but it would need to be fleshed out more. For instance, there should be a way to test that goroutine has terminated. Currently you do this explicitly by passing a channel and waiting for a message.

Re: The case of a leaky goroutine

#33
Just wondering if threads in Rust can suffer such problems?

Background: I am coming from the JS/TS/Node world, and have decided to jump onto a compiled language. I narrowed down my choices to Go and Rust and eventually decided to go with Rust, because it didn't use GC for memory management.

Re: The case of a leaky goroutine

#34
post #9

Didn't Uber have some leaky goroutine detector? I vaguely remember seeing something like that, 5 years ago... Ah yeah it's here. https://github.com/uber-go/goleak

Uber also made something called fx, which is fantastic.

You don't have to use it, but when you do, it helps ensure that you organize your code in a way that becomes very easily testable. It enforces a modular approach to composing together golang services.

Being more easily testable helps prevent bugs, like these leaky goroutines.

Re: The case of a leaky goroutine

#35

Just wondering if threads in Rust can suffer such problems? Background: I am coming from the JS/TS/Node world, and have decided to jump onto a compiled language. I narrowed down my choices to Go and Rust and eventually decided to go with Rust, because it didn't use GC for memory management.

Less likely, but it certainly can. The problem isn't the garbage collector; it's the overall approach to threading. Rust has a different culture that makes problems like this probably less of an issue, but nothing stops you implementing a memory leak.

In fact, since it doesn't have a GC, you can trivially create a memory leak by creating a reference loop... though the ownership checker makes that in itself really difficult, and so it's again less likely to happen than it otherwise would be. At the cost of loops being hard to make even if you want them.

Re: The case of a leaky goroutine

#36
People hate on Haskell async exceptions (with good reason), but one cool thing about them and the Haskell RTS is that you can almost [1] always cancel a thread from the outside. No need for the thread to cooperate like in Golang.

The entire `async` package is built on this. The `race` combinator is an especially cool application.

After doing a big project in Golang, I appreciated this more. We had our fair share of goroutine leaks.

[1] iirc, if the thread is not blocking on a syscall or allocating memory, it will not be yielding to the RTS.

Re: The case of a leaky goroutine

#37
post #19
post #16

Earlier quoted context omitted.

Synchronization such as the whole program is well-defined according to some memory model.

Which is the case for Go: https://go.dev/ref/mem

A quote from your link:

> programmers are strongly encouraged to use appropriate synchronization to avoid data races

Any time you need to "encourage" programmers to do the right thing, you have already failed in your language design.

And I think OP agrees with me here. OP says "static checking of correct synchronization" which is irresponsibly absent from Go.

Re: The case of a leaky goroutine

#38

Just wondering if threads in Rust can suffer such problems? Background: I am coming from the JS/TS/Node world, and have decided to jump onto a compiled language. I narrowed down my choices to Go and Rust and eventually decided to go with Rust, because it didn't use GC for memory management.

Every language can leak memory and threads.

Re: The case of a leaky goroutine

#39

People hate on Haskell async exceptions (with good reason), but one cool thing about them and the Haskell RTS is that you can almost [1] always cancel a thread from the outside. No need for the thread to cooperate like in Golang. The entire `async` package is built on this. The `race` combinator is an especially cool application. After doing a big project in Golang, I appreciated this more. We had our fair share of g…

Rust's futures also get this through being poll-based. You cancel a future by dropping it. Futures compose really easily, so you can combine a bunch of futures in interesting trees of selects and joins and any that are not completed get cleaned up automatically and with little/no overhead when they go out of scope / when the task they're a part of completes. You don't think about them like separate chunks of work, you just think about them like types you can await on and yield a value, and the compiler flattens it all out into a state machine. All of the sync and composition combinators are implemented just in the traits/types of the tokio/futures libraries because the poll/waker abstraction is low level and versatile enough. As long as you don't go out of your way to write bad async code, there's no leaks for the same reasons there's no leaks in Rust code.

It feels a lot like the monadic composition of Haskell even if the means it achieves it are very different.

Re: The case of a leaky goroutine

#40
post #13
post #3

Thread leaks, which happen more frequently due to threaded async abstractions, such as the goroutine, are less often discussed than memory or CPU leaks, but are much more dangerous in multi-tenant container environments. A thread leak can lock up your entire node, including all the control plane processes. A container spec doesn't provide an easy way to control thread/nproc/ulimit limits (you can still do it, but it'…

Isn't this addressed by CPU requests/limits + pid limiting? https://kubernetes.io/docs/concepts/policy/pid-limiting/

It's just more annoying to set up and isn't as widely known, and doesn't work on a per-pod basis.

> PID limiting is a an important sibling to compute resource requests and limits. However, you specify it in a different way: rather than defining a Pod's resource limit in the .spec for a Pod, you configure the limit as a setting on the kubelet. Pod-defined PID limits are not currently supported.

Post reply on HN