Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conc: Better Structured Concurrency for Go

github.com

Post reply on HN
Print view View original

11–20 of 162 posts

Re: Conc: Better Structured Concurrency for Go

#11

Post by stephen123 »

Great project. It seems like channels are just the wrong tool for a lot of concurrency problems. More powerful than needed and easy to get wrong. Lots of nice ways to make go concurrency safer.

The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing results.

Something like go-future helps. https://github.com/stephennancekivell/go-future

Re: Conc: Better Structured Concurrency for Go

#12

Post by alexeldeib »

kevmo314 wrote: post #7

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup > A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable. In go land, this seems desirable. Recoverable errors should be propagated…

A goroutine created inside an http request handler (itself a goroutine) which then panics, by default will crash the whole server, not the single request. The panic could simply be an out of bounds access. That should not crash the whole server.

It’s a logic bug, but you can’t “not panic”. You can trap and recover it though.

Bit orthogonal to OP but relevant to your reasoning.

Re: Conc: Better Structured Concurrency for Go

#13

Post by camdencheek »

chrsig wrote: post #8
openasocket wrote: post #5

I think one of the examples they give is a bit misleading. This func process(stream chan int) { var wg sync.WaitGroup for i := 0; i And func process(stream chan int) { p := pool.New().WithMaxGoroutines(10) for elem := range stream { elem := elem p.Go(func() { handle(elem) }) } p.Wait() } Do slightly different things. The first one has 10 independent, long-lived, go-routines that are all consuming from a single channe…

I haven't looked at the implementation at all, but it is possible that the pool is keeping goroutines alive, and the `Go()` method writes to a single `chan func()` that those goroutines read off of. Which still isn't exactly equivalent, there's still an additional channel read due to the `for elem := range stream {}` loop, and likely an allocation due to the closure.

This is exactly correct. Behavior is equivalent, performance is not. It's probably still not a great example because if reading from a channel already, you're probably better off spawning 10 tasks that read off that channel, but the idea of the example was that it can handle unbounded streams with bounded concurrency.

Re: Conc: Better Structured Concurrency for Go

#14

Post by jerf »

kevmo314 wrote: post #7

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup > A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable. In go land, this seems desirable. Recoverable errors should be propagated…

It is the way of things in an imperative language. If you catch a panic, you are also declaring to the runtime that there is nothing dangling, no locks in a bad state, etc. This is often the case. (Although since I don't think this is a well-understood aspect of what catching a panic means, it is arguably only usually true by a certain amount of coincidence.) But if you don't say that to the runtime, it can't assume it safely and terminating the program, while violent, is arguably either the best option or the only correct option.

Other paradigms, like the Erlang paradigm, can have better behaviors even if a top-level evaluation fails. But in an imperative language, there really isn't anything else you should do. It is arguably one of the Clues (in the "cluestick" sense) that the imperative paradigm is perhaps not the one that should be the base of our technology. But that's a well-debated matter.

Re: Conc: Better Structured Concurrency for Go

#17

Post by camdencheek »

TylerE wrote: post #10
zdragnar wrote: post #4

Is it just me or are the names and descriptions really confusing? i.e. p.WithCollectErrored() configures result pools to only collect results that did not error

Think it's an error on the homepage. If you click through to the actual api doc it makes a lot more sense: " WithCollectErrored configures the pool to still collect the result of a task even if the task returned an error. By default, the result of tasks that errored are ignored and only the error is collected."

Whoops, yep, thanks for pointing it out. Just fixed it

Re: Conc: Better Structured Concurrency for Go

#18

Post by camdencheek »

kevmo314 wrote: post #7

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup > A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable. In go land, this seems desirable. Recoverable errors should be propagated…

> The WaitGroup looks suspiciously like errgroup

I heavily used errgroup before creating conc, so the design is likely strongly influenced by that of errgroup even if not consciously. Conc was partially built to address the shortcomings of errgroup (from my perspective). Probably worth adding a "prior art" section to the README, but many of the ideas in conc are not unique.

> In go land, this seems desirable.

I mostly agree, which is why `Wait()` propagates the panic rather than returning it or logging it. This keeps panics scoped to the spawning goroutine and enables getting stacktraces from both the spawning goroutine and the spawned goroutine, which is quite useful for debugging.

That said, crashing the whole webserver because of one misbehaving request is not necessarily a good tradeoff. Conc moves panics into the spawning goroutine, which makes it possible to do things like catch panics at the top of a request and return a useful error to the caller, even if that error is just "nil pointer dereference" with a stacktrace. It's up to the user to decide what to do with propagated panics.

Re: Conc: Better Structured Concurrency for Go

#19

Post by bxfhjcvu »

alexeldeib wrote: post #12
kevmo314 wrote: post #7

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup > A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable. In go land, this seems desirable. Recoverable errors should be propagated…

A goroutine created inside an http request handler (itself a goroutine) which then panics, by default will crash the whole server, not the single request. The panic could simply be an out of bounds access. That should not crash the whole server. It’s a logic bug, but you can’t “not panic”. You can trap and recover it though. Bit orthogonal to OP but relevant to your reasoning.

[dead]

Re: Conc: Better Structured Concurrency for Go

#20

Post by camdencheek »

stephen123 wrote: post #11

Great project. It seems like channels are just the wrong tool for a lot of concurrency problems. More powerful than needed and easy to get wrong. Lots of nice ways to make go concurrency safer. The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing res…

> run different things in the background and gather the results in different ways

I'd be curious to see an example of the type of task you want to be able to do more safely

Post reply on HN