Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coroutines for Go

research.swtch.com

Post reply on HN
Print view View original

11–20 of 191 posts

Re: Coroutines for Go

#11

Post by wahern »

djha-skin wrote: post #4

I thought that the entire point of green threads was so that I didn't have to use something like Python's `yield` keyword to get nice, cooperative-style scheduling. I thought go's `insert resumes at call points and other specific places` design decision was a very nice compromise. This is allowing access to more and more of the metal. At what point are we just recreating Zig here? What's next? An optional garbage col…

yield and resume here aren't keywords, they're regular variables (referencing regular closures) given those names purely for pedagogical purposes.

The only odd thing here is the creation and use of a cancel callback. I've not done much Go programming, so I don't know whether this is just a performance improvement to more quickly collect dropped iterator state, or it's a requirement because Go does not garbage collect goroutines waiting on channels for which the waiting goroutine has the only reference. In Lua you wouldn't need such a thing because coroutines/threads are GC'd like any other object--once all references are gone they're GC'd even if the last operation was a yield and not a return from the entry function.

Re: Coroutines for Go

#12

Post by gkfasdfasdf »

vaastav wrote: post #7
djha-skin wrote: post #4

I thought that the entire point of green threads was so that I didn't have to use something like Python's `yield` keyword to get nice, cooperative-style scheduling. I thought go's `insert resumes at call points and other specific places` design decision was a very nice compromise. This is allowing access to more and more of the metal. At what point are we just recreating Zig here? What's next? An optional garbage col…

Garbage collector in Go is optional. You can switch off garbage collection by setting the environment variable GOGC=off. More info about GOGC: https://dave.cheney.net/tag/gogc

Ok so garbage collection is optional, how about garbage generation? Is there any way to manually clean up resources if GOCG=off, or will memory usage continue to grow unbounded as new objects are created?

Re: Coroutines for Go

#13

Post by vaastav »

masklinn wrote: post #9
vaastav wrote: post #6

Not sure if this really is required. Most cases in Go are served well by GoRoutines and for yield/resume semantics, 2 blocking channel are enough. This seems to add complexity for the sake of it and not sure it actually adds any new power to Go that already didn't exist.

Goroutines + channels add an enormous amount of overhead. Using them as iterators is basically insane.

Sure but that's what the implementation of the coro in this post uses under the hood. Not sure how this is any better wrt overheads.

Re: Coroutines for Go

#14

Post by freecodyx »

rcme wrote: post #10
masklinn wrote: post #9

Earlier quoted context omitted.

Goroutines + channels add an enormous amount of overhead. Using them as iterators is basically insane.

Why? Channels are already iterable using the range keyword. ch := make(chan int) go func() { for i := 0; i That is very simple.

Race conditions. With coroutines you’re not supposed to deal with race. If i am understanding the motives

Re: Coroutines for Go

#15

Post by vaastav »

gkfasdfasdf wrote: post #12
vaastav wrote: post #7

Earlier quoted context omitted.

Garbage collector in Go is optional. You can switch off garbage collection by setting the environment variable GOGC=off. More info about GOGC: https://dave.cheney.net/tag/gogc

Ok so garbage collection is optional, how about garbage generation? Is there any way to manually clean up resources if GOCG=off, or will memory usage continue to grow unbounded as new objects are created?

Grows unbounded. I wasn't recommending that one should set GOGC=off. Just making a remark that one could should they choose to do so.

EDIT: Sorry, I misunderstood part of your question. The memory grows unbounded unless you call runtime.GC() which triggers garbage collection. But this is a blocking call and essentially block your whole program.

Re: Coroutines for Go

#16

Post by rcme »

freecodyx wrote: post #14
rcme wrote: post #10

Earlier quoted context omitted.

Why? Channels are already iterable using the range keyword. ch := make(chan int) go func() { for i := 0; i That is very simple.

Race conditions. With coroutines you’re not supposed to deal with race. If i am understanding the motives

You can write to a channel concurrently.

Re: Coroutines for Go

#17

Post by geodel »

vaastav wrote: post #13
masklinn wrote: post #9

Earlier quoted context omitted.

Goroutines + channels add an enormous amount of overhead. Using them as iterators is basically insane.

Sure but that's what the implementation of the coro in this post uses under the hood. Not sure how this is any better wrt overheads.

Where did you get this '..it uses same under the hood'? The article clearly says:

..Next I added a direct coroutine switch to the runtime, avoiding channels entirely. That cuts the coroutine switch to three atomic compare-and-swaps (one in the coroutine data structure, one for the scheduler status of the blocking coroutine, and one for the scheduler status of the resuming coroutine), which I believe is optimal given the safety invariants that must be maintained. That implementation takes 20ns per switch, or 40ns per pulled value. This is about 10X faster than the original channel implementation.

Re: Coroutines for Go

#18

Post by jerf »

rcme wrote: post #10
masklinn wrote: post #9

Earlier quoted context omitted.

Goroutines + channels add an enormous amount of overhead. Using them as iterators is basically insane.

Why? Channels are already iterable using the range keyword. ch := make(chan int) go func() { for i := 0; i That is very simple.

"Enormous amount of overhead" is the operative phrase. In general, you want your concurrency operations to be significantly smaller than the payload of the operation. In the case of using channels as iterators with goroutines behind them, it works fine for something like a web page scraper, where the act of fetching a web page is enormously larger than a goroutine switch, but as a generalized iteration mechanism it's unusably expensive because a lot of iteration payloads are very small compared to a channel send.

I've encountered a lot of people who read that on /r/golang and then ask "well why are send operations so expensive", and it's not that. It's that a lot of iteration operations are on the order of single-digit cycles and often very easy to pipeline or predict. No concurrency primitive can keep up with that. A given send operation is generally fairly cheap but there are enough other things that are still an order or two of magnitude cheaper than even the cheapest send operation that if you block all those super cheap operations on a send you're looking at multiple factor of magnitude slowdowns. Such as you would experience with your code.

Re: Coroutines for Go

#19

Post by wahern »

vaastav wrote: post #13
masklinn wrote: post #9

Earlier quoted context omitted.

Goroutines + channels add an enormous amount of overhead. Using them as iterators is basically insane.

Sure but that's what the implementation of the coro in this post uses under the hood. Not sure how this is any better wrt overheads.

> Not sure how this is any better wrt overheads.

At the end he implements an experimental runtime mechanism that permits a goroutine to explicitly switch execution to another goroutine rather than using the generic channel scheduling plumbing.

Re: Coroutines for Go

#20

Post by vaastav »

geodel wrote: post #17
vaastav wrote: post #13

Earlier quoted context omitted.

Sure but that's what the implementation of the coro in this post uses under the hood. Not sure how this is any better wrt overheads.

Where did you get this '..it uses same under the hood'? The article clearly says: ..Next I added a direct coroutine switch to the runtime, avoiding channels entirely. That cuts the coroutine switch to three atomic compare-and-swaps (one in the coroutine data structure, one for the scheduler status of the blocking coroutine, and one for the scheduler status of the resuming coroutine), which I believe is optimal given…

The runtime switch was buried in the last paragraph of the article. All of the code was using goroutines and channels....
Post reply on HN