Conc: Better Structured Concurrency for Go

Post by Groxx » Thu, Jan 12, 2023, 12:14 AM UTC

preseinger wrote: post #40

BreakfastB0b wrote: post #35

Earlier quoted context omitted.

I would agree if it weren’t super easy to cause a panic in go. Index slice out of bounds? panic. Close a channel twice? Panic. Incorrect type assertion? Panic. Dereference nil pointer? Panic. I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potentially modified the application state in an unknown way. It’s like not…

> Index slice out of bounds? panic. Close a channel twice? Panic. Incorrect type assertion? Panic. Dereference nil pointer? Panic. These are all really bad things which should never survive to production code. It is not difficult to detect and prevent them. > I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potenti…

Post by groestl » Thu, Jan 12, 2023, 12:18 AM UTC

preseinger wrote: post #44

camdencheek wrote: post #39

Earlier quoted context omitted.

> panics aren't "goroutine scoped" in terms of their potential impact I'm with ya there. However, there are also many classes of logic errors that are not goroutine-scoped. And there are many panics that do not have impact outside of the goroutine's scope. In my experience, this is true of most panics. In practice, panics happen. They are (almost) always indicative of a bug, and almost always mean there is something…

> In practice, panics happen. I guess this is the crux of the issue. I don't think this is true, or needs to be true. It certainly hasn't been my experience. I think assuming panics are normal will take you down some paths that make it basically impossible to write reliable software. But, to each their own. > I'm accepting the risk that my application is left in an inconsistent state, Inconsistent state makes it impo…

Post by tmerr » Thu, Jan 12, 2023, 12:21 AM UTC

  - I don't want to mess with panics.
  - The default concurrency GOMAXPROCS is almost never what I want.
  - Aggregated errors are almost never what I want. (I haven't read the implementation, but I worry about losing important error codes, or propagating something huge across an RPC boundary).
  - Using it would place a burden on any reader that isn't familiar with the library

Post by camdencheek » Thu, Jan 12, 2023, 12:27 AM UTC

tmerr wrote: post #55

Ahh having a generic pool abstraction that collects results is tempting... nice work. I likely won't use this library though, since - I don't want to mess with panics. - The default concurrency GOMAXPROCS is almost never what I want. - Aggregated errors are almost never what I want. (I haven't read the implementation, but I worry about losing important error codes, or propagating something huge across an RPC boundary…

Post by ilyt » Thu, Jan 12, 2023, 12:36 AM UTC

camdencheek wrote: post #45

ilyt wrote: post #42

func process(stream chan int) { p := pool.New().WithMaxGoroutines(10) for elem := range stream { elem := elem p.Go(func() { handle(elem) }) } p.Wait() I did something similar just with input (optionally output) channel. Close input, goroutines stop, when all of them stop the output is closed [1]. No need to incur function call just to add elements (although I'd imagine go would just inline it so it might not matter e…

Fair criticism. The nice thing about the current API is it works with any input that's iterable (channels, slices, readers, etc.) and any output (callback, channel, append to slice, etc.). In most code I write, I avoid channels because I find them easy to misuse. I used channels in the examples because it's the easiest way to represent "some potentially unbounded stream of input."

    out := MapSlice(
       func(i int) string { return fmt.Sprintf("-=0x%02x=-", i) },
       MapSlice(
          func(i int) int { return i + 1 },
          MapSlice(
             func(i int) int { return i * i },
             GenSlice(10, func(idx int) int { return idx }),
          ),
       ),
    )

    out :=
       WorkerPoolBackgroundClose(
          WorkerPoolBackgroundClose(
             WorkerPoolBackgroundClose(
                GenChanNClose(3, func(idx int) int { return idx + 1 }),
                func(i int) string { return strconv.Itoa(i) },
                4),
             func(s string) string { return ">" + s },
             5,
          ),
          func(s string) string { return " |" + s },
          6)