Earlier quoted context omitted.
But this concept does also solve that problem. Maybe more simply. Rust is renowned for being a total headache with borrowing and lifetimes. Nurseries might be a simpler solution for this. As you say, a corner case. But a common one...
Nurseries and lifetimes are orthogonal concepts. In fact, Rust already has a "nursery", except it's generally called a scoped threadpool. They use lifetimes to ensure the threads can reference stuff on the stack without copying it. So a "nursery" is basically just a scoped threadpool with a 'static lifetime.
Notes on structured concurrency, or: Go statement considered harmful
111–120 of 234 posts
Re: Notes on structured concurrency, or: Go statement considered harmful
#112> As a result, every mainstream concurrency framework I know of simply gives up. If an error occurs in a background task, and you don't handle it manually, then the runtime just... drops it on the floor and crosses its fingers that it wasn't too important. You ought to look into Erlang and Elixir on the BEAM vm/runtime. It's arguably the best example of this kind of concurrency (greenthreading, async) done properly w…
Supervision trees (or sync.WaitGroup in Go) are good tools for achieving the same end result. But it isn't as semantically protected as the article states. In Trio, the property is given by the programs scope.
However, if process creation/termination follows the scoping rules of the program, I have a hunch you run into situations where certain things are not only hard, but outright impossible to express.
Now, the author gambles that this is a good thing and we will eventually find good structural solutions to all the problems. I, on the other hand, is a bit more pessimistic because it has been tried before and found to be lacking.
I wonder how Trio handles error propagation.
Re: Notes on structured concurrency, or: Go statement considered harmful
#113C/POSIX type threads have no language support for indicating what data is shared and which locks protect which data. That's a common cause of trouble. The big question in shared memory concurrency is "who locks what". Most of the bugs in concurrent programs come from ambiguities over that question.
Early attempts to deal with this at the language level included Modula's "monitors", the "rendezvous" in Ada, and Java "synchronized" classes. These all bound the data and its lock together. Rust's locking system does this, and is probably the most successful one so far. (Yes, the functional crowd has their own approaches.)
Go talked a lot about controlling shared memory use. The trouble with goroutines, as Go programmers found out the hard way, was that the "share by communicating, not by sharing" line was bogus. Even the original examples had shared data. But the language didn't provide much support for controlling that sharing.
Python is basically at the C level of sharing control over data, except that the Global Interpreter Lock keeps the low-level data structures from breaking. This prevents Python programs from doing much with multi-core CPUs. Since this is just another thread library for Python, it has the same limitations.
Real concurrency in Python with disjoint data, and without launching a heavy-weight subprocess, would be a big win. But this isn't it.
Re: Notes on structured concurrency, or: Go statement considered harmful
#114Earlier quoted context omitted.
With that approach, you increase the number of threads being used for a single "block" by how deep your "wait" stuff goes. In C# and Python, an async context shares a single thread loop that all joins jump back to. Simply starting a thread and waiting on it isn't suitable in most scenarios, namely web and UI dev. I wish Go had something like this as well.
In Go it doesn't really matter how many "threads" you have because Go has its lightweight threading model (aka coroutines) - creating a new "thread" is very cheap. The reason C#, Python (and Rust), have that model is because they don't have coroutines and starting another thread is very expensive.
Re: Notes on structured concurrency, or: Go statement considered harmful
#115What. This article proposes a "nursery", which is just a wrapped sync.WaitGroup/pthread_join/futures::future::join_all/a reactor that waits for all tasks to terminate/etc. It then uses an exception-like model for error propagation to "solve" error handling (which is fairly easy to handle with a channel). The construct is a decently usable, already applied tool to handle a set of problems, but the article takes the is…
> The "with" example for benefits to not having a "go" statement seem rather bogus, especially seeing that such RAII constructs do not exist in Go (no destructors, remember?).
I don't really understand what you mean by this. You would certainly need a different mechanism in Go than in Python, but I fail to see how that is a criticism of the argument he is making. He has built one implementation in one language, clearly it would look different in other languages.
> Trying to claim that "go" is as terrible as the original "goto" is ignorance of the original problems. Bad use of goto can be a nightmare to track (as the author tried to illustrate), but goroutines do not jump around, they branch from the main goroutine, following normal control flow from there. They are easy to follow, and the language is designed so that you can throw around with them and forget them without them causing you problems.
Why exactly is branching better? With "go" statements, you can easily end up with goroutines that you aren't even aware of floating around doing things that you aren't aware of. His example of calls in external libraries is probably the best illustration. Currently, you could call what you think is a simple function from a library, and end up with a whole host of goroutines you didn't expect floating around, doing things and using resources. Articles about writing good libraries [0] have to make points about how you need to take care of this stuff.
Gotos create problems when you end up surprised about what code is being executed, making tracing execution paths difficult. Branching creates a different set of problems: most crucially that you can have code being executed and not even realize it's happening. The issues are different, but his analogy is really spot on IMO. Both of these things can be solved with code reading and debugging, but the author's whole point is that with an improved abstraction you no longer have to worry about that sort of thing.
> Also, this article is comparing a list of concurrency constructs and one parallism construct (pthread_create—threading.Thread doesn't count as parallism due to GIL) to callbacks, something which have nothing to do with concurrency at all. Very odd.
These are all related concepts. They are about doing different things "at the same time," just with different definitions of what the phrase means. The problems the author are describing are considerably worse in a scenario where actual processes are being spawned, but the nursery pattern is relevant in all of them. It's about control flow, not capabilities. Even using threading with a GIL can create execution paths that you aren't really aware of until you actually look through the code or debug.
Re: Notes on structured concurrency, or: Go statement considered harmful
#116The author appears to have reinvented Communicating Sequential Processes (CSP). http://www.usingcsp.com/cspbook.pdf The (simplified, and as I understand it) gist of concurrency in CSP is that the program is expressed as a series of parallel (PAR) and sequential (SEQ) operations. Everything in a PAR block will run in parallel and all their outputs will be collected and fed as the input to the next SEQ. Everything in a…
goroutines are CSP: https://golang.org/doc/faq#csp The FAQ even mentions Occam, mentioned by another commenter: "Occam and Erlang are two well known languages that stem from CSP. Go's concurrency primitives derive from a different part of the family tree whose main contribution is the powerful notion of channels as first class objects."
Re: Notes on structured concurrency, or: Go statement considered harmful
#117This addresses the wrong problem. The real issue is control over data shared between threads, not control flow. C/POSIX type threads have no language support for indicating what data is shared and which locks protect which data. That's a common cause of trouble. The big question in shared memory concurrency is "who locks what". Most of the bugs in concurrent programs come from ambiguities over that question. Early at…
Can you elaborate on this?
Re: Notes on structured concurrency, or: Go statement considered harmful
#118What. This article proposes a "nursery", which is just a wrapped sync.WaitGroup/pthread_join/futures::future::join_all/a reactor that waits for all tasks to terminate/etc. It then uses an exception-like model for error propagation to "solve" error handling (which is fairly easy to handle with a channel). The construct is a decently usable, already applied tool to handle a set of problems, but the article takes the is…
Agree with everything. I think the misunderstanding here is that goroutines are not classical threads (which would share many problems with `goto`) - they exist somewhere between coroutines and actors (because of channels+select). Actors are an established and mature solution to many headaches surrounding concurrency and parallelism. Goroutines share more in common with method calls (which are a form of branching) an…
Method calls aren't really branching. The code path is still totally linear. You could basically copy paste the code in the method definition in place of its call and get the same outcome (not literally but you know what I mean). For goroutines this is not the case.
I think your points about state ownership are right on, but that's kind of the author's whole point. Right now, there are a lot of things you have to be very conscious of to write good code that executes cleanly using goroutines/threads/etc. That is very much the same as writing good code with gotos. It's 100%, unequivocally possible to write good code using gotos (every control flow structure dijkstra proposed can be represented with them), it just requires a lot of added thought, and the potential for mistakes is much higher.
The author is not proposing any functionality that doesn't already exist, just a new control pattern to reduce the chance of creating problems.
Re: Notes on structured concurrency, or: Go statement considered harmful
#119I'm really growing fond of the async/await abstraction. How do I get this in more C-like languages like Go, Rust, or C++? (I have a bunch of C++ code that I want to call via C ABIs.) I'm intrigued by libdill but it's mysterious enough that I'm scared to include it in my project -- I don't want to risk getting sidetracked by having to debug my concurrency primitives.
Rust is working on exactly this, targeting stabilization later this year: http://aturon.github.io/2018/04/24/async-borrowing/
Very excited!