Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conc: Better Structured Concurrency for Go

github.com

Post reply on HN
Print view View original

1–10 of 162 posts

Re: Conc: Better Structured Concurrency for Go

#2

Post by JamesSwift »

Just took a glance but it seems like this is exactly the kind of project I saw coming out of generics going live. I was really surprised to see how subtly hard go concurrency was to do right when initially learning it. Something like this that formalizes patterns and keeps you from leaking goroutines / deadlocking without fuss is great.

Re: Conc: Better Structured Concurrency for Go

#5

Post by openasocket »

I think one of the examples they give is a bit misleading. This

  func process(stream chan int) {
    var wg sync.WaitGroup
    for i := 0; i
And

  func process(stream chan int) {
    p := pool.New().WithMaxGoroutines(10)
    for elem := range stream {
        elem := elem
        p.Go(func() {
            handle(elem)
        })
    }
    p.Wait()
  }
Do slightly different things. The first one has 10 independent, long-lived, go-routines that are all consuming from a single channel. The second one has the current thread read from the channel and dynamically spawn go-routines. They have the same effect, but different performance characteristics.

Re: Conc: Better Structured Concurrency for Go

#6

Post by PaulKeeble »

JamesSwift wrote: post #2

Just took a glance but it seems like this is exactly the kind of project I saw coming out of generics going live. I was really surprised to see how subtly hard go concurrency was to do right when initially learning it. Something like this that formalizes patterns and keeps you from leaking goroutines / deadlocking without fuss is great.

Its was initially something that surprised me about Go. It was famous for good concurrency and yet compared to many functional languages and contemporary OO languages it had a lot of foot guns. There is a lot of repetition of code that even in languages like Java had long been made common. Go seems to lack obvious concurrency abstractions.

When they announced generics the first thing I did with it was rewrite my common slice parallel algorithm and my limited concurrency pool. It is an obvious area needing improvement for common use cases.

Re: Conc: Better Structured Concurrency for Go

#7

Post by kevmo314 »

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup

> A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable.

In go land, this seems desirable. Recoverable errors should be propagated as return values, not as panics.

Re: Conc: Better Structured Concurrency for Go

#8

Post by chrsig »

openasocket wrote: post #5

I think one of the examples they give is a bit misleading. This func process(stream chan int) { var wg sync.WaitGroup for i := 0; i And func process(stream chan int) { p := pool.New().WithMaxGoroutines(10) for elem := range stream { elem := elem p.Go(func() { handle(elem) }) } p.Wait() } Do slightly different things. The first one has 10 independent, long-lived, go-routines that are all consuming from a single channe…

I haven't looked at the implementation at all, but it is possible that the pool is keeping goroutines alive, and the `Go()` method writes to a single `chan func()` that those goroutines read off of.

Which still isn't exactly equivalent, there's still an additional channel read due to the `for elem := range stream {}` loop, and likely an allocation due to the closure.

Re: Conc: Better Structured Concurrency for Go

#9

Post by camdencheek »

Hi! Author here. Conc is the result of generalizing and cleaning up an internal package I wrote for use within Sourcegraph. Basically, I got tired of rewriting code that handled panics, limited concurrency, and ensured goroutine cleanup. Happy to answer questions or address comments.

Re: Conc: Better Structured Concurrency for Go

#10

Post by TylerE »

zdragnar wrote: post #4

Is it just me or are the names and descriptions really confusing? i.e. p.WithCollectErrored() configures result pools to only collect results that did not error

Think it's an error on the homepage.

If you click through to the actual api doc it makes a lot more sense: " WithCollectErrored configures the pool to still collect the result of a task even if the task returned an error. By default, the result of tasks that errored are ignored and only the error is collected."

Post reply on HN