Distributed coroutines with a native Python extension and Dispatch

This is part 2 of https://news.ycombinator.com/item?id=39696423

I understand that it's a very interesting engineering problem - to create distrubuted coroutines and make them work in Python, but I'm not sure I understand why I would want to use this concept in our projects. I understand the idea of coroutines as a way to increase throughput (the CPU isn't stalled waiting for I/O to finish) - Go already has that. However, what are the advantages of rescheduling an already running coroutine on a different machine? Shouldn't it be the job of the load balancer, to choose the least busy machine, before any coroutine is run? Isn't it expensive to serialize coroutines and transfer them between machines? Also, I wonder what observability is like. If a coroutine crashes, what its stacktrace will look like?

> However, what are the advantages of rescheduling an already running coroutine on a different machine?

Your original machine might not have the required computational resources to run the job quickly enough.

> Shouldn't it be the job of the load balancer, to choose the least busy machine, before any coroutine is run?

It is not always simple or possible to know, when what part of a computation will be finished and as such cannot easily be perfectly planned. If you mean a load balancer as in traefik or similar, then that entails serializing your intermediate results or writing your code in a specific way, so that computation can be split using a load balancer.

> Isn't it expensive to serialize coroutines and transfer them between machines?

Probably, but not taking advantage of idle cores on another machine might be more expensive (in terms of time needed to finish the computation).

Also, I wonder what observability is like. If a coroutine crashes, what its stacktrace will look like?

No idea, have not used it.

This kind of thing is what Erlang excels at. Serializing functions and their entire environments is a difficult to solve problem. I think in Python it probably means moving into a whole different space of types and objects, because Python has not been developed with such a thing in mind from the start, while Erlang has.

Dispatch takes a different approach with different trade-offs, but you're right that capturing the local program scope brings interesting challenges.

Serializing file handles doesn't work, but in our experience, programs rarely run into constructs where this becomes a problem, and when it happens, there are mitigation measures that are usually easy to implement (small restructure of the program, capturing resource metadata to reconstruct them later, etc...).

We have a few features on the roadmap to help mitigate the security implications as well, including allowing users to store their program state in a S3 bucket that they own. Our scheduler can operate with only metadata about the program, so splitting the two can be an effective model to mitigate those risks.

>Probably, but not taking advantage of idle cores on another machine might be more expensive (in terms of time needed to finish the computation).

If there are idle cores on another machine, why not just point the load balancer to schedule new jobs/coroutines on those idle machines? Why reschedule existing coroutines on another machine if they run just fine on the current node? (i.e. the node is at full capacity, and it's fine) I can see that the problem can arise if a coroutine wants to spawn new coroutines, and if we schedule them on the same node which is at full capacity, they can end up waiting for a while before they are run... It makes sense to schedule new jobs on different machines, but why reschedule existing coroutines? You have to serialize/deserialize stuff and send over the wire, and there are several gotchas as explained in the article (pickling arbitrary objects doesn't sound very reliable/safe, and judging from the article, can break in future versions of Python), plus there's a lot of magic involved which will probably make it harder to investigate things in production... I'd personally just stick with local coroutine sheduler + global load balancer which picks nodes for new jobs + coroutines, when created, only receive plain values as arguments (string, int, float and basic structures) to be reliably serializable and so that it was transparent what's going on... (i.e. do NOT store internal coroutine state, assume they are transient). Maybe I don't understand the idea.