Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed coroutines with a native Python extension and Dispatch

stealthrocket.tech

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Distributed coroutines with a native Python extension and Dispatch

#11

Post by achille-roussel »

kgeist wrote: post #4

I understand that it's a very interesting engineering problem - to create distrubuted coroutines and make them work in Python, but I'm not sure I understand why I would want to use this concept in our projects. I understand the idea of coroutines as a way to increase throughput (the CPU isn't stalled waiting for I/O to finish) - Go already has that. However, what are the advantages of rescheduling an already running…

> Go already has that. However, what are the advantages of rescheduling an already running coroutine on a different machine?

Say your coroutine is performing a transactional operation with multiple steps, but for some reason the second step starts getting rate limits from an upstream API, you need to enter a retry loop to reschedule the operation a bit later. What if your program stops then? Without capturing the state somewhere, it remains volatile in memory and the operation would be lost when the program stops.

This scenario occurs in a wide range of systems, from simple async jobs for email delivery to complex workflows with fan-out and fan-in.

> Isn't it expensive to serialize coroutines and transfer them between machines?

We only capture the local scope of the call stack, so the amount of data that we need to record is pretty small and stays in the order of magnitude of the data being processed (e.g., the request payload).

> I wonder what observability is like. If a coroutine crashes, what its stacktrace will look like?

We maintain stack traces across await points of coroutine calls, even if the concurrent operations are scheduled on different program instances. When an error occurs, the stack traces show the entire code path that led to the error.

There's definitely a lot of opportunities for creating powerful introspection tools, but since Dispatch is just code, you can also use your typical metrics, logs, and traces to instrument your distributed coroutines.

Re: Distributed coroutines with a native Python extension and Dispatch

#12

Post by kgeist »

achille-roussel wrote: post #11
kgeist wrote: post #4

I understand that it's a very interesting engineering problem - to create distrubuted coroutines and make them work in Python, but I'm not sure I understand why I would want to use this concept in our projects. I understand the idea of coroutines as a way to increase throughput (the CPU isn't stalled waiting for I/O to finish) - Go already has that. However, what are the advantages of rescheduling an already running…

> Go already has that. However, what are the advantages of rescheduling an already running coroutine on a different machine? Say your coroutine is performing a transactional operation with multiple steps, but for some reason the second step starts getting rate limits from an upstream API, you need to enter a retry loop to reschedule the operation a bit later. What if your program stops then? Without capturing the sta…

>We only capture the local scope of the call stack, so the amount of data that we need to record is pretty small and stays in the order of magnitude of the data being processed

What if the local stack references a deep object hierarchy? The classical "I wanted a banana, but got the gorilla holding the banana and the entire jungle"

>What if your program stops then? Without capturing the state somewhere, it remains volatile in memory and the operation would be lost when the program stops.

You persist coroutines on disk?

Re: Distributed coroutines with a native Python extension and Dispatch

#13

Post by achille-roussel »

kgeist wrote: post #10
zelphirkalt wrote: post #6

Earlier quoted context omitted.

> However, what are the advantages of rescheduling an already running coroutine on a different machine? Your original machine might not have the required computational resources to run the job quickly enough. > Shouldn't it be the job of the load balancer, to choose the least busy machine, before any coroutine is run? It is not always simple or possible to know, when what part of a computation will be finished and as…

>Probably, but not taking advantage of idle cores on another machine might be more expensive (in terms of time needed to finish the computation). If there are idle cores on another machine, why not just point the load balancer to schedule new jobs/coroutines on those idle machines? Why reschedule existing coroutines on another machine if they run just fine on the current node? (i.e. the node is at full capacity, and…

Distributed coroutines are a primitive to express transactional workflows that may last longer than the initial request/response that triggered it (think any form of async operation). While the distribution allows effective use of compute resources, capturing the state of coroutines and their progress is the key addition that enables the execution of workflows and guarantees completion.

A load balancer can help distribute new jobs across a fleet, but even the shortest of jobs can become "long running" when it hits timeouts, rate limits, and other transient errors. You quickly need a scheduler to effectively orchestrate the retries without DDoS-ing your systems, and need to keep track of the state to carry jobs to completion.

Combine a scheduler (like Dispatch) with a primitive like distributed coroutines, and you've got a powerful foundation to create distributed applications of all kinds without seeing complexity skyrocket.

Re: Distributed coroutines with a native Python extension and Dispatch

#14

Post by kgeist »

achille-roussel wrote: post #13
kgeist wrote: post #10

Earlier quoted context omitted.

>Probably, but not taking advantage of idle cores on another machine might be more expensive (in terms of time needed to finish the computation). If there are idle cores on another machine, why not just point the load balancer to schedule new jobs/coroutines on those idle machines? Why reschedule existing coroutines on another machine if they run just fine on the current node? (i.e. the node is at full capacity, and…

Distributed coroutines are a primitive to express transactional workflows that may last longer than the initial request/response that triggered it (think any form of async operation). While the distribution allows effective use of compute resources, capturing the state of coroutines and their progress is the key addition that enables the execution of workflows and guarantees completion. A load balancer can help distr…

OK, from what I understand, it's similar to what we do as well, except Dispatch adds magic while we do it all manually. We have an event-based system: instead of await points, we fire events which are stored inside an AMQP broker. The broker has N consumers on different nodes which take new jobs as they arrive. Retries/circuit breakers etc. are added manually (via a Go library), and if a job/event handler fails, it's readded back to the AMQP queue (someone else will process it later). Inside event handlers/job processors we also enjoy Go's builtin local scheduler (so I/O calls do not block entire cores).

I can see the benefit that with Dispatch, logic is simpler to read/to write as just ordinary functions, while in our approach, we have to scatter it around various event handlers/job processors. However, I still like that in our approach, event handlers/job processors are entirely stateless (the only state is jobs/event payloads), I've found it to be good for scalability and reliability + easier to reason about, compared to passing around internal coroutine state.

Re: Distributed coroutines with a native Python extension and Dispatch

#15

Post by sitkack »

faizshah wrote: post #5

I feel like temporal’s approach of replaying the computation is probably better than trying to serialize the running computation. Serializing the running coroutine gets ugly as is shown here with things like file handles and making pickle a central part of your compute platform is a little scary from an AppSec pov. That being said, I like the idea and the blog post is wonderfully written.

With the right abstractions, running computation is just data.

Re: Distributed coroutines with a native Python extension and Dispatch

#16

Post by richieartoul »

achille-roussel wrote: post #8
faizshah wrote: post #5

I feel like temporal’s approach of replaying the computation is probably better than trying to serialize the running computation. Serializing the running coroutine gets ugly as is shown here with things like file handles and making pickle a central part of your compute platform is a little scary from an AppSec pov. That being said, I like the idea and the blog post is wonderfully written.

Dispatch takes a different approach with different trade-offs, but you're right that capturing the local program scope brings interesting challenges. Serializing file handles doesn't work, but in our experience, programs rarely run into constructs where this becomes a problem, and when it happens, there are mitigation measures that are usually easy to implement (small restructure of the program, capturing resource me…

Storing the data in the customer's S3 bucket is a really good idea IMO. Means they just have to trust you to run the coordination reliably, but not necessarily to keep their data secure which is a much easier bar to clear.

Re: Distributed coroutines with a native Python extension and Dispatch

#18

Post by achille-roussel »

kgeist wrote: post #14
achille-roussel wrote: post #13

Earlier quoted context omitted.

Distributed coroutines are a primitive to express transactional workflows that may last longer than the initial request/response that triggered it (think any form of async operation). While the distribution allows effective use of compute resources, capturing the state of coroutines and their progress is the key addition that enables the execution of workflows and guarantees completion. A load balancer can help distr…

OK, from what I understand, it's similar to what we do as well, except Dispatch adds magic while we do it all manually. We have an event-based system: instead of await points, we fire events which are stored inside an AMQP broker. The broker has N consumers on different nodes which take new jobs as they arrive. Retries/circuit breakers etc. are added manually (via a Go library), and if a job/event handler fails, it's…

Yes, that sounds very similar indeed. We've launched Dispatch because this is a universal problem that engineering teams end up having to reinvent over and over.

Dispatch can also handle the "one-off" jobs you describe, where you don't need to track the coroutine state. In a way, it's a subset/special case of the distributed coroutine (just like functions are a special case of coroutines with no yield point).

Re: Distributed coroutines with a native Python extension and Dispatch

#19

Post by achille-roussel »

kgeist wrote: post #12
achille-roussel wrote: post #11

Earlier quoted context omitted.

> Go already has that. However, what are the advantages of rescheduling an already running coroutine on a different machine? Say your coroutine is performing a transactional operation with multiple steps, but for some reason the second step starts getting rate limits from an upstream API, you need to enter a retry loop to reschedule the operation a bit later. What if your program stops then? Without capturing the sta…

>We only capture the local scope of the call stack, so the amount of data that we need to record is pretty small and stays in the order of magnitude of the data being processed What if the local stack references a deep object hierarchy? The classical "I wanted a banana, but got the gorilla holding the banana and the entire jungle" >What if your program stops then? Without capturing the state somewhere, it remains vol…

A local variable can end up referencing a large part of the program state. Some of this can be detected and automated to avoid putting unnecessary burden on the developers, but it's also part of our roadmap for Dispatch to create tools that help developers understand how their application will behave when it starts getting distributed.

This is why these solutions have often seen the creation of a new programming language as a necessary step to avoid those issues altogether. We think it's important to bring those capabilities to mainstream languages, we can't always go back from scratch with a new language.

Regarding where the state is stored, we keep everything in S3 and have plans to allow users to provide their own object store if they need to keep the application state in a storage medium that they own.

Re: Distributed coroutines with a native Python extension and Dispatch

#20

Post by paulddraper »

sitkack wrote: post #15
faizshah wrote: post #5

I feel like temporal’s approach of replaying the computation is probably better than trying to serialize the running computation. Serializing the running coroutine gets ugly as is shown here with things like file handles and making pickle a central part of your compute platform is a little scary from an AppSec pov. That being said, I like the idea and the blog post is wonderfully written.

With the right abstractions, running computation is just data.

They abstraction being the universe.
Post reply on HN