Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conc: Better Structured Concurrency for Go

github.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: Conc: Better Structured Concurrency for Go

#61

Post by klabb3 »

camdencheek wrote: post #9

Hi! Author here. Conc is the result of generalizing and cleaning up an internal package I wrote for use within Sourcegraph. Basically, I got tired of rewriting code that handled panics, limited concurrency, and ensured goroutine cleanup. Happy to answer questions or address comments.

I literally started drafting my own structured concurrency proposal for Go 2 today, due to exactly the same frustrations you mention. Such a coincidence, and thanks for writing this lib. I will most certainly use it.

Please could you tell me if you have any thoughts on how to integrate these ideas into the language?

One thing I think should be solved (and that appears not addressed by your lib?) is the function coloring problem of context. I would really, really love if context was implicit and universal cancel/deadline mechansim, all the way down to IO. That way, only parts of the chain that NEED to use the context would be affected (which is a minority in practice). Any thoughts on that?

Finally, I think somebody should create a more visionary and coherent proposal for improved concurrency in Go2, because individual proposals on Go’s issue tracker are shut down quickly due to not making enough sense in isolation - and status quo bias. It’s a shame because I really love Go and want to see reduced footguns and boilerplate – especially with concurrency often being the largest source of complexity in a project. Please find my email in profile if you’re interested to pursue this further.

Thanks again.

Re: Conc: Better Structured Concurrency for Go

#62

Post by groestl »

Groxx wrote: post #51
preseinger wrote: post #40

Earlier quoted context omitted.

> Index slice out of bounds? panic. Close a channel twice? Panic. Incorrect type assertion? Panic. Dereference nil pointer? Panic. These are all really bad things which should never survive to production code. It is not difficult to detect and prevent them. > I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potenti…

> These are all really bad things which should never survive to production code. It is not difficult to detect and prevent them. This is equivalent to saying "out of bounds memory writes are not difficult to detect and prevent in C code". Like actually equivalent (possibly worse), not just "well if you squint they look similar". Of course it's not hard most of the time . Being perfect is beyond hard though. And if yo…

> Sometimes shutting down every piece of your software if that happens is the correct choice, and sometimes it's so far beyond reasonable that it's ludicrous to argue in favor of "every panic is an abort".

Very much this. And even for the same project: in some cases, I'm a fan of employing a quite strict error handling policy in dev environments (crash and burn) and using a more lenient approach in prod (elevated log level). In my experience, this can result in a robust product. Most importantly, this means the decision is not even made by the application programmer, sometimes it's a config thing.

Re: Conc: Better Structured Concurrency for Go

#63

Post by erik_seaberg »

preseinger wrote: post #40
BreakfastB0b wrote: post #35

Earlier quoted context omitted.

I would agree if it weren’t super easy to cause a panic in go. Index slice out of bounds? panic. Close a channel twice? Panic. Incorrect type assertion? Panic. Dereference nil pointer? Panic. I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potentially modified the application state in an unknown way. It’s like not…

> Index slice out of bounds? panic. Close a channel twice? Panic. Incorrect type assertion? Panic. Dereference nil pointer? Panic. These are all really bad things which should never survive to production code. It is not difficult to detect and prevent them. > I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potenti…

x == y can panic if interface values contain incomparable fields in unexported nested structs, how would I check for that? Should we let it become a query of death and bet thousands of peers’ jobs on it never happening?

Re: Conc: Better Structured Concurrency for Go

#64

Post by jasonhansel »

preseinger wrote: post #44
camdencheek wrote: post #39

Earlier quoted context omitted.

> panics aren't "goroutine scoped" in terms of their potential impact I'm with ya there. However, there are also many classes of logic errors that are not goroutine-scoped. And there are many panics that do not have impact outside of the goroutine's scope. In my experience, this is true of most panics. In practice, panics happen. They are (almost) always indicative of a bug, and almost always mean there is something…

> In practice, panics happen. I guess this is the crux of the issue. I don't think this is true, or needs to be true. It certainly hasn't been my experience. I think assuming panics are normal will take you down some paths that make it basically impossible to write reliable software. But, to each their own. > I'm accepting the risk that my application is left in an inconsistent state, Inconsistent state makes it impo…

> An account value that previously had balance = 0 may now have balance = 1000. Is this acceptable risk?

Your entire web app process crashes due to a panic every time a request triggers an extremely rare edge case. A hacker discovers this and uses it to conduct a DoS attack. Is this acceptable risk?

Re: Conc: Better Structured Concurrency for Go

#65

Post by [deleted] »

morelisp wrote: post #24
alexeldeib wrote: post #12

Earlier quoted context omitted.

A goroutine created inside an http request handler (itself a goroutine) which then panics, by default will crash the whole server, not the single request. The panic could simply be an out of bounds access. That should not crash the whole server. It’s a logic bug, but you can’t “not panic”. You can trap and recover it though. Bit orthogonal to OP but relevant to your reasoning.

> simply be an out of bounds access If you have any care for quality at all, there's nothing "simple" about your invariants being violated.

[deleted]

Re: Conc: Better Structured Concurrency for Go

#66

Post by foobiekr »

kevmo314 wrote: post #7

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup > A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable. In go land, this seems desirable. Recoverable errors should be propagated…

if you've ever had to deal with a zombie JVM where an uncaught exception eventually put the whole application into a state where it wasn't, oh, hoovering the tens of thousands of temporary files it was creating because the vacuumer thread died undetected, the Go behavior is _extremely_ attractive.

Re: Conc: Better Structured Concurrency for Go

#67

Post by tjdevries »

Groxx wrote: post #46

https://github.com/sourcegraph/conc/blob/main/iter/iter.go#L... // Map applies f to each element of input, returning the mapped result. func Map[T, R any](input []T, f func(*T) R) []R { res := make([]R, len(input)) ForEachIdx(input, func(i int, t *T) { res[i] = f(t) }) return res } Seems a little silly to farm that off to a custom func when you could just write the for-loop, but it's probably fine / may be no differe…

I'm not sure what you don't like? Sorry, must just be missing your point.

Re: Conc: Better Structured Concurrency for Go

#68

Post by stephen123 »

ilyt wrote: post #43
stephen123 wrote: post #11

Great project. It seems like channels are just the wrong tool for a lot of concurrency problems. More powerful than needed and easy to get wrong. Lots of nice ways to make go concurrency safer. The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing res…

>The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing results. Do you have any examples ? About only that I can think of is "parse something to a bunch of different types" and that can be solved easily enough. What do you mean by "reusing results" ?…

The main difference is that reading from channels will block if its empty where Futures, return the same value.

Written more concisely.

f := New(func() Type{return t}) v := g.Get() w := g.Get()

Re: Conc: Better Structured Concurrency for Go

#69

Post by ilyt »

stephen123 wrote: post #68
ilyt wrote: post #43

Earlier quoted context omitted.

>The problem that bothers me (and isnt in Conc), is how hard it is to run different things in the background and gather the results in different ways. Particularly when you start doing those things conditionally and reusing results. Do you have any examples ? About only that I can think of is "parse something to a bunch of different types" and that can be solved easily enough. What do you mean by "reusing results" ?…

The main difference is that reading from channels will block if its empty where Futures, return the same value. Written more concisely. f := New(func() Type{return t}) v := g.Get() w := g.Get()

I wish Go had macros a'la Rust, it would be possible to write the whole thing in so much nicer way.

Re: Conc: Better Structured Concurrency for Go

#70

Post by groestl »

preseinger wrote: post #32
alexeldeib wrote: post #28

Earlier quoted context omitted.

So you've never written code with a bug? There are other ways to panic in go - concurrent map writes, nil pointer dereference. I'm not saying it should happen, but best practice would be a defensive posture especially when it's effectively zero cost, not hoping for the best.

Crashing the program _is_ the defensive posture. Panics -- concurrent map writes or nil pointer dereferences or almost anything else -- usually mean the program state has become invalid. You can't treat them like errors.

> usually mean the program state has become invalid.

_some_ part of a program's state has become invalid. Probably a variable on the stack. In web servers, almost always a part that becomes very irrelevant as soon as the current request ends. In databases, the same is true at transaction boundaries. In shells it's the REPL. Exit(1) in any of those deep down the stack? No thank you, wouldn't touch that software with a 10 foot pole.

Post reply on HN