Live data from Hacker News

Notes on structured concurrency, or: Go statement considered harmful

vorpus.org

81–90 of 234 posts

Re: Notes on structured concurrency, or: Go statement considered harmful

#81

The author appears to have reinvented Communicating Sequential Processes (CSP). http://www.usingcsp.com/cspbook.pdf The (simplified, and as I understand it) gist of concurrency in CSP is that the program is expressed as a series of parallel (PAR) and sequential (SEQ) operations. Everything in a PAR block will run in parallel and all their outputs will be collected and fed as the input to the next SEQ. Everything in a…

goroutines are CSP: https://golang.org/doc/faq#csp

The FAQ even mentions Occam, mentioned by another commenter:

"Occam and Erlang are two well known languages that stem from CSP. Go's concurrency primitives derive from a different part of the family tree whose main contribution is the powerful notion of channels as first class objects."

Re: Notes on structured concurrency, or: Go statement considered harmful

#82

The author appears to have reinvented Communicating Sequential Processes (CSP). http://www.usingcsp.com/cspbook.pdf The (simplified, and as I understand it) gist of concurrency in CSP is that the program is expressed as a series of parallel (PAR) and sequential (SEQ) operations. Everything in a PAR block will run in parallel and all their outputs will be collected and fed as the input to the next SEQ. Everything in a…

It's not immediately clear whether or not the reified nursery object breaks this correspondance.

Re: Notes on structured concurrency, or: Go statement considered harmful

#83

Earlier quoted context omitted.

par is "select" effectively. While that isn't 100% the case, it is true.

Huh? I thought "select" was "ALT"??

'go' is PAR with different syntax (Everything after the 'go' line is implicitly one branch of the PAR)

'select' is ALT

Re: Notes on structured concurrency, or: Go statement considered harmful

#84

Earlier quoted context omitted.

Knowing whether a background task has spawned is very different from not being able to follow the control flow (goto potentially jumping to an entirely different function body). Now, while Go is designed mostly for you to not care about goroutines, there are some corner cases where one must know if a resource is used by anything, such as the chase of when you wish to close a file handle. However, I'd argue that this…

But this concept does also solve that problem. Maybe more simply. Rust is renowned for being a total headache with borrowing and lifetimes. Nurseries might be a simpler solution for this. As you say, a corner case. But a common one...

Nurseries and lifetimes are orthogonal concepts. In fact, Rust already has a "nursery", except it's generally called a scoped threadpool. They use lifetimes to ensure the threads can reference stuff on the stack without copying it. So a "nursery" is basically just a scoped threadpool with a 'static lifetime.

Re: Notes on structured concurrency, or: Go statement considered harmful

#85

Earlier quoted context omitted.

A process has much higher overhead (both in time and space), more complex communications and is not always possible in the first place. It also adds complexity to the build pipeline. These quickly adds up to being not worth it over a task or thread. You wouldn't spawn a process for the rendering/input/physics/network/job/loader threads of a game engine for one. You can't spawn processes from a web app either. Threads…

> A process has much higher overhead (both in time and space), more complex communications and is not always possible in the first place. This can change if OSes start optimizing for the proposed convention. > It also adds complexity to the build pipeline. At least in C/C++ it doesn't. But again, this is also easily solvable if we want to. > You wouldn't spawn a process for the rendering/input/physics/network/job/loa…

threads were actually invented for these sorts of use cases, as "lightweight processes".

If OSs change again, sure, the way we program will change again.

At the moment, using processes is too resource intensive, that's exactly why we have threads in the first place. And I don't understand what benefit you are suggesting using processes over threads would have.

Re: Notes on structured concurrency, or: Go statement considered harmful

#86
post #76

Article persuasive prima facie and arguments plausible. I've had to reinvent a structured method of managing threads a number of times, unfortunately. Author is a PhD student, which bodes well for not reinventing wheels dumbly. Therefore, I look forward to the lit review of other concurrency & parallelism work through the last 40 years, which this writeup notably lacks (author mentions his stack of papers to review).…

> Author is a PhD student, which bodes well for not reinventing wheels dumbly.

Except that's exactly what the author did. They just reinvented scoped threadpools.

Re: Notes on structured concurrency, or: Go statement considered harmful

#87
The main culprit to me in Golang seems to be channels, not goroutines. If your workflow essentially is defined by a mesh of channels and goroutines, it's hard to reason or understand.

I have no direct practical knowledge of Golang, but working on a large application that used BlockingQueue for concurrent communication and one which extensively used services buses for communication - both were hard to understand and reason about flow.

After some years with Scala Futures I'd say they work well and reason well. They can be seen as normal function calls returning Future instead of another 'container'.

They reflect the black box mentioned in the article, with one way in and one way out (e.g. when a method returns Future[_]).

The point about error handling: We use Option,Seq.empty on read error handling, Validation on create/write and Either on side effects (like sending mail).

(yes, they are still leaky abstractions e.g. when debugging, but work fine most of the time)

Re: Notes on structured concurrency, or: Go statement considered harmful

#88

The author appears to have reinvented Communicating Sequential Processes (CSP). http://www.usingcsp.com/cspbook.pdf The (simplified, and as I understand it) gist of concurrency in CSP is that the program is expressed as a series of parallel (PAR) and sequential (SEQ) operations. Everything in a PAR block will run in parallel and all their outputs will be collected and fed as the input to the next SEQ. Everything in a…

Speaking of CSP, more broadly process algebras (a.k.a. process calculus) generally have such a "parallel composition" operator.

Also, CSP genuine inter-process communication primitive is a _multi-way_ rendezvous which can synchronise an arbitrary number of process, possibly more than two. This is called "interactions" in chapter 2 of Tony Hoare CSP book [1], and channels are built on top of these interactions in chapter 4.

This multi-way synchronization is also present e.g. in the LOTOS specification language, which is an ISO standard. The CADP [2] verification toolbox offers various tools like model-checker to verify LOTOS programs, and also to generate executables. For those who know how special/weird the LOTOS syntax is, the CADP folks also develops the LNT language which looks much more like Ada/Pascal.

[1] http://usingcsp.com/cspbook.pdf [2] http://cadp.inria.fr/

Re: Notes on structured concurrency, or: Go statement considered harmful

#89

What. This article proposes a "nursery", which is just a wrapped sync.WaitGroup/pthread_join/futures::future::join_all/a reactor that waits for all tasks to terminate/etc. It then uses an exception-like model for error propagation to "solve" error handling (which is fairly easy to handle with a channel). The construct is a decently usable, already applied tool to handle a set of problems, but the article takes the is…

You realise that makes you sound exactly like an old Fortran programmer refusing to give up goto? A for loop is just a wrapped goto! ;-)

I think the author proposes a useful way to structure multi-threading. The comparison to goto isn't perfect and he needs to play fast and loose with some terms to keep the analogy working but he makes a good point.

I'm not convinced yet that the nursery pattern should be the only allowed way to start a thread but saying "you should use it unless you have a good reason not to" is a good provocation to get the discussion going.

Re: Notes on structured concurrency, or: Go statement considered harmful

#90
So, my initial thought when reading was "yeah, it's async/await". But it's subtly different than that though.

You're free to spawn parallel tasks in async/await -- you just use Promise.all or Future.sequence, or whatever your language provides to compose them into a larger awaitable.

Nurseries seem to go a step beyond this by reifying the scheduling scope as the eponymous nursery object. This means that you have a new choice when the continuation of your async task happens: a nursery pass in from some ancestor of the call tree. My gut says that this offers similar power as problematic fire-and-forget async tasks, but takes away the ability to truly forget about them.

My guess is that, in practice, you end up with some root level nursery in your call stack to account for this. But account for it you must! And while the overwhelming sentiment in these comments is pretty dismissive, I'd caution against downplaying the significance of this. It's basically like checked exceptions or monadic error handling.

I also think about how this maps to task or IO monad models of concurrency. It seems like there's an inversion of control. Rather than returning the reification of a task to be scheduled later, the task takes the reification of a runtime, upon which to schedule itself. I'm not sure what the ramifications of this are. Maybe it would help with the virality of async return values [1], but at the cost of the virality of nursery arguments.

Lastly, one thing this article nails is the power of being able to reason about continuation of control flow. Whether or not nurseries have merit as a novel construct, this article still has a lot of educational use by making this argument very clearly. Even if the author is wrong about nurseries being "the best", it sets a compelling standard that all control mechanisms--async or not--should have to explain themselves against.

I do have a couple questions:

- In the real world, would library APIs begin to get clogged with the need for a nursery on which to run an async task? Think async logging or analytics libraries.

- Would usages similar to long-running tasks that receive async messages be compatible? I'm thinking of usages of the actor model or channel model that implement dynamic work queues.

- Does this increase the hazard presented by non-halting async tasks?

[1] http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

Post reply on HN