Notes on structured concurrency, or: Go statement considered harmful • Hacker News Board

Re: Notes on structured concurrency, or: Go statement considered harmful

#161

Post by elcritch » Wed, Apr 25, 2018, 11:40 PM UTC

jlouis wrote: post #112

leshow wrote: post #57

> As a result, every mainstream concurrency framework I know of simply gives up. If an error occurs in a background task, and you don't handle it manually, then the runtime just... drops it on the floor and crosses its fingers that it wasn't too important. You ought to look into Erlang and Elixir on the BEAM vm/runtime. It's arguably the best example of this kind of concurrency (greenthreading, async) done properly w…

Somewhat agree. Supervision trees (or sync.WaitGroup in Go) are good tools for achieving the same end result. But it isn't as semantically protected as the article states. In Trio, the property is given by the programs scope. However, if process creation/termination follows the scoping rules of the program, I have a hunch you run into situations where certain things are not only hard, but outright impossible to expre…

Not sure what you mean by semantically protected? OTP supervisor trees enforce the constraints that are specified in the supervisor’s start function. It’s semantics are defined by the OTP library and function scopes. As you mention there are times to break out of the strict (supervisor) pattern if needed/wanted and start processes manually. Otherwise the default behaviors provide a nice set of limited behaviors that have proved useful across a broad spectrum of situations.

Really what TFA’s discussing seems much more akin to OTP than raw Erlang. Or more specifically a subset of it. Occasionally I wish there were a few more OTP supervisor behaviors but nothing that’s a show stopper. Not familiar with Go’s WaitGroup but I haven’t seen it used much in code I’ve read.

Re: Notes on structured concurrency, or: Go statement considered harmful

#162

Post by shalabhc » Wed, Apr 25, 2018, 11:41 PM UTC

"our call stack has become a tree"

This is a really useful property to have and reason about.

Instead of several independent coroutines with arbitrarily overlapping lifetimes, we can now think of all coroutines as organized in a single hierarchy with properly nested lifetimes.

The function call stack becomes a call tree - each branch is a concurrent execution.

Re: Notes on structured concurrency, or: Go statement considered harmful

#163

Post by kllrnohj » Wed, Apr 25, 2018, 11:47 PM UTC

Animats wrote: post #152

kllrnohj wrote: post #147

Earlier quoted context omitted.

Except shared data doesn't give you a little bit better performance, it gives you massively better performance. Or, in some cases, it's the only way to get usable performance at all. Now what you could do is break objects down into annotated types. Consider immutable vs. mutable in combination with thread-unsafe, thread-compatible, and thread-safe. Immutable data that's not thread-unsafe you can share freely across t…

I proposed something like that for Python in 2010.[1] Immutable objects could be shared. Mutable objects had to either be unshared, or a subclass of an object that enforced synchronization. Python's Little Tin God didn't like it. Mostly because I proposed to freeze the code of the program once the second thread started. That takes away much of the dynamism he insists on. Might be worth looking at again. The separatio…

Python seems like an odd place to try and shove this into. Both due to its heavy object mutability & dynamic nature in combination with the GIL making heavy threading of python code a waste of everyone's time anyway.

There aren't many languages with the concept of ownership or moving at all, and trying to retro-fit that is probably going to be not a good experience for anyone involved.

Rust is largely there, in yet another thing it does well, but if you don't want that something like C++ would be probably a better place to try it. There you at least have move & ownership as a language concept already.

Re: Notes on structured concurrency, or: Go statement considered harmful

#164

Post by glorioustao » Wed, Apr 25, 2018, 11:48 PM UTC

go statement is just another form of explicit process creation, fork/join pattern. What the author suggested is just similar to cobegin/coend -- implicit process creation.

cobegin/coend are limited to properly nested graphs, however fork/join can express arbitrary functional parallelism (any process flow graph) [1]

Yes, for graceful error handling it needs to form some sort of process tree or ATC(Asynchronous transfer of control), which is implemented in Erlang/OTP and ada programming.

[1] http://www.ics.uci.edu/~dillenco/compsci143a/notes/ch02.pdf

Re: Notes on structured concurrency, or: Go statement considered harmful

#165

Post by wybiral » Thu, Apr 26, 2018, 12:01 AM UTC

arghwhat wrote: post #136

marcus_holmes wrote: post #36

Earlier quoted context omitted.

But this concept does also solve that problem. Maybe more simply. Rust is renowned for being a total headache with borrowing and lifetimes. Nurseries might be a simpler solution for this. As you say, a corner case. But a common one...

It actually does not solve the problem of lifetimes. It only solves a very isolated instance of it. For example, if we take the problem of an os.File, some library might store the reference, and use it unexpectedly in a later function call after you called Close. This presents the exact same issue as a goroutine holding it. (I honestly have no other examples of this issue than files/connections being closed premature…

I've never encountered a situation in Go where I'm passing file handles around like this.

In practice you'd probably have one goroutine manage the resource and its lifespan then other code would communicate with that goroutine using channels.

Re: Notes on structured concurrency, or: Go statement considered harmful

#166

Post by wokwokwok » Thu, Apr 26, 2018, 12:03 AM UTC

joshuamorton wrote: post #28

shadowmint wrote: post #7

This is very very long to explain a very simple pair of ideas: 1) You should be able to declare scoped blocks that mandate execution of all tasks started in that block ends when the scope ends. 2) This is fundamentally superior to all other forms of concurrency. I get it; this is basically what async/await gets you, but conceptually you can spawn parallel tasks inside an awaited block, and know, absolutely that all o…

>this is distinct from a normal awaited block which will execute tasks inside it sequentially, awaiting each one in turn I believe that python's `async for` construct (which may be stolen from c#, but I'm not sure) does this, and another user mentioned Promise.all which in javascript I believe allows the child promises to resolve concurrently. In both cases, these are for concurrency, not parallelism.

The point of the construct is not explicitly waiting for child tasks/threads/whatever, but that as a formal restriction, you cannot leave the scoped block with unresolved tasks.

I don't believe either of the constructs you've mentioned do this, and I'm not aware of any that do.

The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code.

It doesn't matter if you've called Promise.all or not.

Also, regarding parallelism: Obviously async/await are for that; this is basically pitching an equivalent construct for parallel processing.

I think this is novel, frankly, and the alternatives being thrown around by people are by people who didn't read the article.

Re: Notes on structured concurrency, or: Go statement considered harmful

#167

Post by zzzcpan » Thu, Apr 26, 2018, 12:05 AM UTC

I know people already said it, but causality, resource cleanup and error handling are all solved with Actor model in a more general, more flexible and more reliable way than nurseries.

If you think reasoning about concurrency is hard, try testing and modelling it, especially for something distributed. This is where naive ideas about concurrency should start to fail and a need in solid foundation arise.

Re: Notes on structured concurrency, or: Go statement considered harmful

#168

Post by Animats » Thu, Apr 26, 2018, 12:27 AM UTC

kllrnohj wrote: post #163

Animats wrote: post #152

Earlier quoted context omitted.

I proposed something like that for Python in 2010.[1] Immutable objects could be shared. Mutable objects had to either be unshared, or a subclass of an object that enforced synchronization. Python's Little Tin God didn't like it. Mostly because I proposed to freeze the code of the program once the second thread started. That takes away much of the dynamism he insists on. Might be worth looking at again. The separatio…

Python seems like an odd place to try and shove this into. Both due to its heavy object mutability & dynamic nature in combination with the GIL making heavy threading of python code a waste of everyone's time anyway. There aren't many languages with the concept of ownership or moving at all, and trying to retro-fit that is probably going to be not a good experience for anyone involved. Rust is largely there, in yet a…

When I wrote that, Rust didn't exist, C++ didn't use move semantics much, and Python was more important than it is now.

Re: Notes on structured concurrency, or: Go statement considered harmful

#169

Post by joshuamorton » Thu, Apr 26, 2018, 12:27 AM UTC

wokwokwok wrote: post #166

joshuamorton wrote: post #28

Earlier quoted context omitted.

>this is distinct from a normal awaited block which will execute tasks inside it sequentially, awaiting each one in turn I believe that python's `async for` construct (which may be stolen from c#, but I'm not sure) does this, and another user mentioned Promise.all which in javascript I believe allows the child promises to resolve concurrently. In both cases, these are for concurrency, not parallelism.

The point of the construct is not explicitly waiting for child tasks/threads/whatever, but that as a formal restriction , you cannot leave the scoped block with unresolved tasks. I don't believe either of the constructs you've mentioned do this, and I'm not aware of any that do. The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code. It doesn't matter if you've called…

>The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code.

Well of course, setTimeout isn't async/await based, its callback based. If instead you only had this api, you could make that formal assertion:

    await promiseSetTimeout(time).then(...);

And in fact you can write promiseSetTimeout today!

So basically, async/await provides this as long as you only use async/await. In JS you can't formally assert this because there are functions that subvert the normal control flow, but if you outlaw those, you absolutely can.

Re: Notes on structured concurrency, or: Go statement considered harmful

#170

Post by tomtheelder » Thu, Apr 26, 2018, 12:36 AM UTC

arghwhat wrote: post #132

tomtheelder wrote: post #115

Earlier quoted context omitted.

It really is a whole wrapped up group of existing constructs, but that doesn't mean there isn't merit in it. The author mentions how this was central to Dijkstra's proposal for structured programming: "And now that Dijkstra understood the problem, he was able to solve it. Here's his revolutionary proposal: we should stop thinking of if/loops/function calls as shorthands for goto, but rather as fundamental primitives…

> but that doesn't mean there isn't merit in it Of course not. It also isn't original, or a "silver bullet" that I find relevant to general concurrent programming. > I don't really understand what you mean by this. I was trying to keep my rant a bit short, but my point is that the only practical example for a post that puts a lot of effort into complaining about a core Go construct (it's the title) does not even remo…

Ah I get what you mean about the use of go as an example.

Also, I think when the author refers to callbacks they are referring to callbacks in the JS sense, that is to say callbacks in the context of an event loop type environment. The author certainly wouldn't care about other types of callbacks, since they do not create the split execution paths that he is describing.

The distinction you are trying to draw between concurrency and parallelism, while totally valid, isn't particularly relevant here. The problems the author is describing are almost entirely to do with developers writing and understanding code, and they generally apply to both parallel and concurrent systems. The user experience of writing parallel and concurrent code can often feel extremely similar, so the tools the author is describing will tend to apply in both situations.