Live data from Hacker News

Conc: Better Structured Concurrency for Go

github.com

81–90 of 162 posts

Re: Conc: Better Structured Concurrency for Go

#81
post #75

Earlier quoted context omitted.

> nice work Thanks! > The default concurrency GOMAXPROCS is almost never what I want. FWIW, the default concurrency has been changed to "unlimited" since the 0.1.0 release. > Aggregated errors are almost never what I want. Out of curiousity, what do you want? There is an option to only keep the first error, and it's possible to unwrap the error to an array of errors that compose it if you just want a slice of errors.…

> FWIW, the default concurrency has been changed to "unlimited" since the 0.1.0 release. Nice! Will that end up on Github? > Out of curiousity, what do you want Most often I want to return just the first error. Some reasons: (1) smaller error messages passed across RPC boundaries (2) original errors can be inspected as intended (e.g. error codes) (3) when the semantics are to cancel after the first error, the errors…

Thanks for the feedback!

> Will that end up on Github?

It's already there! I just haven't cut a release since the change.

> Most often I want to return just the first error.

In many cases, I do too, which is why (*pool).WithFirstError() exists :)

> original errors can be inspected as intended

If you're using errors.Is() or errors.As(), error inspection should still work as expected.

> Often goroutine overhead is negligible and I would bound concurrency in dumber ways

Yes, definitely. And that's what I've always done too. However, I've found it's surprisingly easy to get subtly wrong, especially when modifying code I didn't write, and even more especially if I want to propagate panics (which I do, though that seems to be a somewhat controversial opinion in this thread). Conc is intended to be a well-known pattern that I don't have to think about too much when using it.

> I think(?) conc does that too, but it could use documentation

It does! I'll update the docs to make that more clear.

Re: Conc: Better Structured Concurrency for Go

#82
Go's standard library has some really great code, and some really terrible code (error--particularly related to wrapping/unwrapping/is/as). I'll definitely look at this to see if it gets our team's services better stability/performance.

Re: Conc: Better Structured Concurrency for Go

#83
post #77

Earlier quoted context omitted.

x == y can panic if interface values contain incomparable fields in unexported nested structs, how would I check for that? Should we let it become a query of death and bet thousands of peers’ jobs on it never happening?

Is this really the case? Can you link to anything or an example on go.dev/play/ ? I can find a mention of "cmp.Equal" having that behavior, but that's just a third-party package panic.

It's true, but you'd never really write code like this

https://go.dev/play/p/r9NkQb6bQTx

Re: Conc: Better Structured Concurrency for Go

#84
post #7

The WaitGroup looks suspiciously like errgroup, which even has the .WithMaxGoroutines() functionality: https://pkg.go.dev/golang.org/x/sync/errgroup > A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable. In go land, this seems desirable. Recoverable errors should be propagated…

I started to write Go recently and I very much prefer panics to errors. For example in web apps. Panic produces nice stack trace. Panic bubbles automatically, I can reduce amount of code significantly. Errors are awful and panics allow to write code without losing sanity. It's like good old exceptions. And if I really need it, I can catch panic and act as needed, e.g. return HTTP code or whatever.

You should rarely reach for panic. Panics are not like exceptions, really. It's very frowned upon for panics to pass API boundaries. Use errors always, unless the state of the world is irrecoverably broken.

It's difficult to come up with examples of when this may be the case. Often it's a "you're holding it wrong" kind of thing. For example, a common idiom is wrapping something like

    func Parse(s string) (Foo, error)
to be usable in contexts where error handling isn't possible, such as variable initialization.

    func Must(f Foo, err error) Foo {
        if err != nil {
            panic(err)
        }
        return f
    }
which could be used at the global level like so:

    var someFoo = Must(Parse("hey"))
This is acceptable because this code is static and will either panic at boot always or never.

Re: Conc: Better Structured Concurrency for Go

#85
post #14

Earlier quoted context omitted.

It is the way of things in an imperative language. If you catch a panic, you are also declaring to the runtime that there is nothing dangling, no locks in a bad state, etc. This is often the case. (Although since I don't think this is a well-understood aspect of what catching a panic means, it is arguably only usually true by a certain amount of coincidence.) But if you don't say that to the runtime, it can't assume…

I don’t think this is really a question of whether your code is imperative, since Haskell code will terminate just as surely as Go code if you try to access an array element out of range. (Haskell’s lazy evaluation just makes it a bit harder to catch, since you need to force evaluation of the thunk within the catch statement, and it’s far too easy to end up passing your thunk to somebody who won’t catch the exception…

It is a fundamental problem with the imperative paradigm because of the equivalent of:

    lock.Take()
    thingThatMayCrash()
    lock.Release()
The imperative/structured paradigm is that those statements are evaluated in order. While this is not the only way of writing Go, it is legal Go, and the runtime must account for it. Paradigms for which that is not fundamentally true have different options available to them. One of those options is still to crash.

However, "access an array element out of range" isn't the question. The question is, what can the language do in the face of any exception? Haskell, perhaps ironically, doesn't avail itself of the opportunity to do something useful about it. You can implement a wide variety of mechanisms that are safe to have exceptions in even in the face of concurrency, but at the base language, it is essentially as exception unsafe as Go, exactly as you observe. (And you can create "exception-safe wrappers" in Go as well, as easily as having an intermediate function that does something with panics, then invokes some code. Nominally this is unsafe because the code being invoked really ought to have a guarantee that the user has written it to be safe in this usage; in practice it works fairly well at significant scales due to a combination of other things beyond the scope of this already-large reply.) This causes significant practical stress within the community and probably would be in the Top 5 wishlist for a lot of people as to something a sequel language would fix. (It is very annoying that the supposedly "pure" code "head []", supposedly of type "[a] -> a", throws an IO exception.) Just as in Haskell you can write safe code, you can write safe code in Go here as well.

    lock.Take()
    defer lock.Release()
    thingThatMayCrash()
is every bit as safe as anything you can write in Haskell. (It may not compose as well, because "locks" are arguably the least composable primitive ever devised by computer science, but that's an entirely different discussion.)

Thus it is not a coincidence that when I named a language that has a different paradigm that allows it to systematically recover from this sort of fault, I named Erlang, not Haskell. Erlang has no locks. (At the Erlang level. NIFs may do their own thing but they are basically a form of "unsafe" and for language analysis should be treated as such.) Since it has no locks it can't blow up the way the Go code can and leave dangling locks. Processes get access to resources that need to be cleaned up through the IMHO somewhat confusingly named "ports" system, which can be conceived of as wrapping up sockets and files and other such resources behind other Erlang processes, and then there's a linking system that says "if this process crashes, send this message to that other process or crash it also". So an Erlang process can safely crash, release all associated resources, and the runtime may continue on with the assurance that there's no dangling locks or other concurrency things half done.

In fact in my opinion Erlang isn't even a particularly "functional" language. This is idiosyncratic and I don't deny it. A better way of understanding Erlang is that it was built around this functionality existing, and the ports system and immutable terms are the tools that were used to accomplish this goal, with the fact that the result looked sort of "functional" being an accident. So I wouldn't necessarily highlight "functional" languages in general as being natively good at this sort of handling. I think it's pretty clear it could be added to a Haskell++ without much effort but it is not something that merely "being functional" automatically gets you out of the box.

(Historically, Erlang actually descends from Prolog. While all the logic functionality is stripped out, the heritage is very visible. You can't really call it a "logic language" since it can do no logic, but in my opinion it isn't in the functional stream of languages either. Your mileage will vary; like I said I don't deny this is an idiosyncratic take, but for what it's worth, it's one from someone who used Erlang professionally for many years and knows Haskell fairly well too, so it's not an entirely uninformed one.)

Re: Conc: Better Structured Concurrency for Go

#86
post #61

Hi! Author here. Conc is the result of generalizing and cleaning up an internal package I wrote for use within Sourcegraph. Basically, I got tired of rewriting code that handled panics, limited concurrency, and ensured goroutine cleanup. Happy to answer questions or address comments.

I literally started drafting my own structured concurrency proposal for Go 2 today, due to exactly the same frustrations you mention. Such a coincidence, and thanks for writing this lib. I will most certainly use it. Please could you tell me if you have any thoughts on how to integrate these ideas into the language? One thing I think should be solved (and that appears not addressed by your lib?) is the function color…

> I would really, really love if context was implicit and universal cancel/deadline mechansim, all the way down to IO.

I don't think this is an improvement. Implicit behavior is difficult to identify and reason about. The only criticism of context that seems valid to me, aside from arbitrary storage being a huge antipattern, is that it was added long after nearly the entire standard library was authored, and it's usage is still relatively sparse.

We can agree that concurrency is difficult to use correctly, but since the introduction of generics it's much easier to wrap channels and goroutines.

Aside, in my experience if you're worried about boilerplate you're almost always looking at the problem wrong and optimizing for convenience over simplicity.

Re: Conc: Better Structured Concurrency for Go

#87
post #72

Earlier quoted context omitted.

x == y can panic if interface values contain incomparable fields in unexported nested structs, how would I check for that? Should we let it become a query of death and bet thousands of peers’ jobs on it never happening?

don't do that? this kind of thing is why deepcompare exists to begin with

[deleted]

Re: Conc: Better Structured Concurrency for Go

#88
If your code panics, your process should probably just crash (unless you're abusing panics to pass state around to yourself, but that's another story). The overall program state could be invalid and continuing to run may be dangerous. Your program needs to be able to recover from an unexpected termination anyway.

Not recovering panics goes extra for generic packages that are executing user code. You have no idea how badly broken things are.

Re: Conc: Better Structured Concurrency for Go

#89

Earlier quoted context omitted.

I started to write Go recently and I very much prefer panics to errors. For example in web apps. Panic produces nice stack trace. Panic bubbles automatically, I can reduce amount of code significantly. Errors are awful and panics allow to write code without losing sanity. It's like good old exceptions. And if I really need it, I can catch panic and act as needed, e.g. return HTTP code or whatever.

You should rarely reach for panic. Panics are not like exceptions, really. It's very frowned upon for panics to pass API boundaries. Use errors always, unless the state of the world is irrecoverably broken. It's difficult to come up with examples of when this may be the case. Often it's a "you're holding it wrong" kind of thing. For example, a common idiom is wrapping something like func Parse(s string) (Foo, error)…

Errors don't have stack traces so they're unusable. And I see nothing wrong with handling panics. They're the same exceptions. I got SQL exception, I panic, handler catches panic, logs its stacktrace and returns HTTP 500. Awesome and no boring error handling.

Re: Conc: Better Structured Concurrency for Go

#90
post #77

Earlier quoted context omitted.

Is this really the case? Can you link to anything or an example on go.dev/play/ ? I can find a mention of "cmp.Equal" having that behavior, but that's just a third-party package panic.

It's true, but you'd never really write code like this https://go.dev/play/p/r9NkQb6bQTx

The problem also affects structs that happen to have a private map or cache or callback anywhere within.

https://go.dev/play/p/uP-vjpvuhku

Post reply on HN