You gotta appreciate good branding.
Conc: Better Structured Concurrency for Go
41–50 of 162 posts
Re: Conc: Better Structured Concurrency for Go
#42 func process(stream chan int) {
p := pool.New().WithMaxGoroutines(10)
for elem := range stream {
elem := elem
p.Go(func() {
handle(elem)
})
}
p.Wait()
This
func mapStream(
in chan int,
out chan int,
f func(int) int,
) {
s := stream.New().WithMaxGoroutines(10)
for elem := range in {
elem := elem
s.Go(func() stream.Callback {
res := f(elem)
return func() { out WorkerPool[T1, T2 any](input chan T1, output chan T2, worker func(T1) T2, concurrency int)
* [1] https://github.com/XANi/goneric/blob/master/worker.go#L92
Re: Conc: Better Structured Concurrency for Go
#43Great project. It seems like channels are just the wrong tool for a lot of concurrency problems. More powerful than needed and easy to get wrong. Lots of nice ways to make go concurrency safer. The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing res…
>The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing results.
Do you have any examples ? About only that I can think of is "parse something to a bunch of different types" and that can be solved easily enough. What do you mean by "reusing results" ?
> Something like go-future helps. https://github.com/stephennancekivell/go-future
f := future.New(func() string {
return "value"
})
value := f.Get()
f := Async(func() Type{return t})
v := Re: Conc: Better Structured Concurrency for Go
#44Earlier quoted context omitted.
> That said, crashing the whole webserver because of one misbehaving request is not necessarily a good tradeoff. Conc moves panics into the spawning goroutine, which makes it possible to do things like catch panics at the top of a request and return a useful error to the caller, even if that error is just "nil pointer dereference" with a stacktrace. It's up to the user to decide what to do with propagated panics. The…
> panics aren't "goroutine scoped" in terms of their potential impact I'm with ya there. However, there are also many classes of logic errors that are not goroutine-scoped. And there are many panics that do not have impact outside of the goroutine's scope. In my experience, this is true of most panics. In practice, panics happen. They are (almost) always indicative of a bug, and almost always mean there is something…
> In practice, panics happen.
I guess this is the crux of the issue. I don't think this is true, or needs to be true. It certainly hasn't been my experience. I think assuming panics are normal will take you down some paths that make it basically impossible to write reliable software. But, to each their own.
> I'm accepting the risk that my application is left in an inconsistent state,
Inconsistent state makes it impossible to reason about your program's execution or outcomes. An account value that previously had balance = 0 may now have balance = 1000. Is this acceptable risk?
Re: Conc: Better Structured Concurrency for Go
#45func process(stream chan int) { p := pool.New().WithMaxGoroutines(10) for elem := range stream { elem := elem p.Go(func() { handle(elem) }) } p.Wait() I did something similar just with input (optionally output) channel. Close input, goroutines stop, when all of them stop the output is closed [1]. No need to incur function call just to add elements (although I'd imagine go would just inline it so it might not matter e…
Fair criticism. The nice thing about the current API is it works with any input that's iterable (channels, slices, readers, etc.) and any output (callback, channel, append to slice, etc.). In most code I write, I avoid channels because I find them easy to misuse. I used channels in the examples because it's the easiest way to represent "some potentially unbounded stream of input."
Re: Conc: Better Structured Concurrency for Go
#46https://github.com/sourcegraph/conc/blob/main/iter/iter.go#L...
// Map applies f to each element of input, returning the mapped result.
func Map[T, R any](input []T, f func(*T) R) []R {
res := make([]R, len(input))
ForEachIdx(input, func(i int, t *T) {
res[i] = f(t)
})
return res
}
Let's go see what ForEach does, to make sure: https://github.com/sourcegraph/conc/blob/main/iter/iter.go#L...
// ForEachIdx is the same as ForEach except it also provides the
// index of the element to the callback.
func ForEachIdx[T any](input []T, f func(int, *T)) {
numTasks := runtime.GOMAXPROCS(0)
numInput := len(input)
if numTasks > numInput {
// No more tasks than the number of input items.
numTasks = numInput
}
var idx atomic.Int64
// Create the task outside the loop to avoid extra closure allocations.
task := func() {
i := int(idx.Add(1) - 1)
for ; i Re: Conc: Better Structured Concurrency for Go
#47Earlier quoted context omitted.
So you've never written code with a bug? There are other ways to panic in go - concurrent map writes, nil pointer dereference. I'm not saying it should happen, but best practice would be a defensive posture especially when it's effectively zero cost, not hoping for the best.
Crashing the program _is_ the defensive posture. Panics -- concurrent map writes or nil pointer dereferences or almost anything else -- usually mean the program state has become invalid. You can't treat them like errors.
Taking my own example - you can both monitor for panics per http request, while preserving SLO for other requests server by the same process.
Crashing the program is strictly worse imo, since there are other concurrent requests which will now fail for no reason.
I agree unhandled panicking makes sense sometimes. It’s context dependent.
Program state is a good argument, but I think it really depends.
Re: Conc: Better Structured Concurrency for Go
#48https://github.com/sourcegraph/conc/blob/main/iter/iter.go#L... // Map applies f to each element of input, returning the mapped result. func Map[T, R any](input []T, f func(*T) R) []R { res := make([]R, len(input)) ForEachIdx(input, func(i int, t *T) { res[i] = f(t) }) return res } Seems a little silly to farm that off to a custom func when you could just write the for-loop, but it's probably fine / may be no differe…
[deleted]
Re: Conc: Better Structured Concurrency for Go
#49You gotta appreciate good branding.
[deleted]
Re: Conc: Better Structured Concurrency for Go
#50Earlier quoted context omitted.
> panics aren't "goroutine scoped" in terms of their potential impact I'm with ya there. However, there are also many classes of logic errors that are not goroutine-scoped. And there are many panics that do not have impact outside of the goroutine's scope. In my experience, this is true of most panics. In practice, panics happen. They are (almost) always indicative of a bug, and almost always mean there is something…
> In practice, panics happen. I guess this is the crux of the issue. I don't think this is true, or needs to be true. It certainly hasn't been my experience. I think assuming panics are normal will take you down some paths that make it basically impossible to write reliable software. But, to each their own. > I'm accepting the risk that my application is left in an inconsistent state, Inconsistent state makes it impo…
Since defers run during panics for exactly this reason, no. You can in fact guarantee that is not the case.
Runtime-safety "panics" in Go, like concurrently modifying and iterating a map that can lead to other memory being corrupted, tend to abort the whole process immediately and not be suppress-able panics.