Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The case of a leaky goroutine

brainbaking.com

Post reply on HN
Print view View original

21–30 of 87 posts

Re: The case of a leaky goroutine

#21

Post by atombender »

bakul wrote: post #17

If Go allowed something like "handle = go foo()", goroutine could be automatically terminated when handle goes out of scope or becomes dead and is garbage collected. You can also use handle to cancel a goroutine etc. Go designers specifically avoided this model (of having a goroutine "id") for reasons I don't remember any more (may be to avoid making them heavier weight?) but this would be one way to stop leaky gorou…

It's because just terminating a goroutine isn't safe with respect to I/O and defer chains.

I think Go has the internal plumbing to theoretically support this, though it might require inserting checks more often. Another way would be to make contexts first-class and automatically insert context checks even when not done (e.g. selects). And also all I/O has to be cancellable.

I suspect Go's designers prefers the current way in which cancellation is explicit.

Re: The case of a leaky goroutine

#22

Post by atombender »

I wish Go recorded the timestamp of goroutine and let you access them.

An app I work on recently had a bug where goroutines would slowly build up over time. Turns out the bug is in the Growthbook SDK [1]. We can monitor the number of goroutines, but having a large number of goroutines waiting in the location that gets stuck is normal — we can only see such a problem over multiple days, in that the minimum value slowly goes up.

If Go could tell you the timestamp of the oldest goroutines as part of the pprof dump, we could have an alert, and it would work for any such leak.

[1] https://github.com/growthbook/growthbook-golang/pull/28

Re: The case of a leaky goroutine

#23

Post by paulddraper »

jerf wrote: post #2

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g... There's a library for it: https://github.com/sourcegraph/conc But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new l…

Structure concurrency is to concurrency what structured program control are to program control.

I.e. unstructured concurrency is like GOTOs. Not necessarily wrong, but certainly nerve-wracking.

Re: The case of a leaky goroutine

#24

Post by bakul »

atombender wrote: post #21
bakul wrote: post #17

If Go allowed something like "handle = go foo()", goroutine could be automatically terminated when handle goes out of scope or becomes dead and is garbage collected. You can also use handle to cancel a goroutine etc. Go designers specifically avoided this model (of having a goroutine "id") for reasons I don't remember any more (may be to avoid making them heavier weight?) but this would be one way to stop leaky gorou…

It's because just terminating a goroutine isn't safe with respect to I/O and defer chains. I think Go has the internal plumbing to theoretically support this, though it might require inserting checks more often. Another way would be to make contexts first-class and automatically insert context checks even when not done (e.g. selects). And also all I/O has to be cancellable. I suspect Go's designers prefers the curren…

> just terminating a goroutine isn't safe with respect to I/O and defer chains.

Agreed. The idea is to panic() if a goroutine has to be forcibly terminated due to GC, instead of a slow leak. Requires more thought though.

Re: The case of a leaky goroutine

#25

Post by codethief »

I don't have any experience with Go but to my untrained eyes this looks very much like a general problem that I've noticed with coroutines in other languages, e.g. JavaScript or Python¹: Coroutines are so lightweight that people tend to "fire & forget" them, when in reality coroutines take up memory and can easily leak. One should keep track of them and garbage-collect them but last time I checked there wasn't a great out-of-the-box solution for that.

¹ Same thing in frameworks like RxJS, where observers in some sense take on the role of coroutines.

Re: The case of a leaky goroutine

#26

Post by [deleted] »

jerf wrote: post #2

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g... There's a library for it: https://github.com/sourcegraph/conc But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new l…

[deleted]

Re: The case of a leaky goroutine

#27

Post by duped »

hedora wrote: post #4
lxe wrote: post #3

Thread leaks, which happen more frequently due to threaded async abstractions, such as the goroutine, are less often discussed than memory or CPU leaks, but are much more dangerous in multi-tenant container environments. A thread leak can lock up your entire node, including all the control plane processes. A container spec doesn't provide an easy way to control thread/nproc/ulimit limits (you can still do it, but it'…

My #1 complaint about Rust is that leaking a future is safe. It means the compiler can’t check for async coroutine leaks, and it breaks the borrow checker’s ability to say “nothing else has a reference to this any more”. Anyway, we’re using golang for some stuff at work, and holy crap, I forgot how terrible it was to work in high level languages that don’t statically check for correct synchronization. If C++-style co…

> My #1 complaint about Rust is that leaking a future is safe.

Do you mean futures that aren't polled to completion, tasks that aren't joined, or literal memory leaks that happen to own futures?

Re: The case of a leaky goroutine

#28

Post by kbolino »

lxe wrote: post #3

Thread leaks, which happen more frequently due to threaded async abstractions, such as the goroutine, are less often discussed than memory or CPU leaks, but are much more dangerous in multi-tenant container environments. A thread leak can lock up your entire node, including all the control plane processes. A container spec doesn't provide an easy way to control thread/nproc/ulimit limits (you can still do it, but it'…

Goroutines in a single process map onto a fixed number of threads. Even if you have goroutine leaks, you should not have thread leaks. Your program may deadlock or run out of memory, but it will not take the whole system down (at least, not in this way).

Re: The case of a leaky goroutine

#29

Post by masklinn »

bakul wrote: post #24
atombender wrote: post #21

Earlier quoted context omitted.

It's because just terminating a goroutine isn't safe with respect to I/O and defer chains. I think Go has the internal plumbing to theoretically support this, though it might require inserting checks more often. Another way would be to make contexts first-class and automatically insert context checks even when not done (e.g. selects). And also all I/O has to be cancellable. I suspect Go's designers prefers the curren…

> just terminating a goroutine isn't safe with respect to I/O and defer chains. Agreed. The idea is to panic() if a goroutine has to be forcibly terminated due to GC, instead of a slow leak. Requires more thought though.

So the idea is to randomly take down the program when the GC runs?

Re: The case of a leaky goroutine

#30

Post by kbolino »

samatman wrote: post #20

A lot of these problems come from accepting a line in the OP, "A Goroutine is essentially a coroutine". The rest of the sentence is "...that maps onto green threads that map onto real native threads on your OS in an NxM way". This is not a coroutine at all, calling them goroutines was a clever hacker pun along the lines of "GNU's Not Unix". If you treat a preëmptively-scheduled primitive as though it's a cooperativel…

I was under the impression that goroutines were cooperative but that they yield pretty aggressively (on blocking channel operations, disk and network I/O, cgo calls, and certain syscalls). What makes you think they're preemptive?
Post reply on HN