Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed coroutines with a native Python extension and Dispatch

stealthrocket.tech

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Distributed coroutines with a native Python extension and Dispatch

#21

Post by achille-roussel »

gigatexal wrote: post #17

I don't understand how this is priced -- or how it works. Why do I need to sign up with my Github account? Why do I need an API key?

We've just launched in developed preview, so no pricing yet, but this is coming!

A good starting point to answer your other questions is the "getting started" section in our documentation https://docs.dispatch.run/dispatch/getting-started

If you have more questions, join our Discord! We're always happy to chat and get feedback on what's difficult to grasp.

Re: Distributed coroutines with a native Python extension and Dispatch

#22

Post by achille-roussel »

zelphirkalt wrote: post #6
kgeist wrote: post #4

I understand that it's a very interesting engineering problem - to create distrubuted coroutines and make them work in Python, but I'm not sure I understand why I would want to use this concept in our projects. I understand the idea of coroutines as a way to increase throughput (the CPU isn't stalled waiting for I/O to finish) - Go already has that. However, what are the advantages of rescheduling an already running…

> However, what are the advantages of rescheduling an already running coroutine on a different machine? Your original machine might not have the required computational resources to run the job quickly enough. > Shouldn't it be the job of the load balancer, to choose the least busy machine, before any coroutine is run? It is not always simple or possible to know, when what part of a computation will be finished and as…

Your comment on this being solved by inventing new programming languages like Erlang is right on point.

Our take is that distributed coroutines can be brought to general-purpose programming languages like Python so that engineering teams can adopt these features incrementally in their existing applications instead of adding a whole new language or framework.

In my opinion, the value is in being able to reuse the software tools and processes you're familiar with, major shifts are rarely the right call.

Re: Distributed coroutines with a native Python extension and Dispatch

#24

Post by achille-roussel »

halfcat wrote: post #23

When would one want to use this, compared to, say, Celery?

Celery is one of those popular solutions that is commonly used. However, it comes with a complex framework for describing workflows, which is often difficult to master, especially for younger engineers.

With Dispatch, you just write code, as if you would write a local program, and our scheduler can manage the distribution and state of the execution. Testing the code especially gets really simple.

Having a hosted scheduler also means that you don't need to deploy any extra infrastructure like RabbitMQ or Redis.

Re: Distributed coroutines with a native Python extension and Dispatch

#26

Post by gigatexal »

achille-roussel wrote: post #21
gigatexal wrote: post #17

I don't understand how this is priced -- or how it works. Why do I need to sign up with my Github account? Why do I need an API key?

We've just launched in developed preview, so no pricing yet, but this is coming! A good starting point to answer your other questions is the "getting started" section in our documentation https://docs.dispatch.run/dispatch/getting-started If you have more questions, join our Discord! We're always happy to chat and get feedback on what's difficult to grasp.

For me this is DOA unless it can be run locally without having to call into some hosted API in the cloud.

I took a gander at the source I'll wait till/if there's ever a 100% local version of this. But good luck all the same!

Re: Distributed coroutines with a native Python extension and Dispatch

#27

Post by achille-roussel »

gigatexal wrote: post #26
achille-roussel wrote: post #21

Earlier quoted context omitted.

We've just launched in developed preview, so no pricing yet, but this is coming! A good starting point to answer your other questions is the "getting started" section in our documentation https://docs.dispatch.run/dispatch/getting-started If you have more questions, join our Discord! We're always happy to chat and get feedback on what's difficult to grasp.

For me this is DOA unless it can be run locally without having to call into some hosted API in the cloud. I took a gander at the source I'll wait till/if there's ever a 100% local version of this. But good luck all the same!

I guess this is your lucky day, because you can already run it locally!

Take a look at the "Local Testing" section in the README of dispatch-py https://github.com/stealthrocket/dispatch-py?tab=readme-ov-f...

TLDR: python -m dispatch.test http://127.0.0.1:8000

This is just a mock implementation but it lets you test your code and experiment with the solution.

Re: Distributed coroutines with a native Python extension and Dispatch

#28

Post by gigatexal »

achille-roussel wrote: post #27
gigatexal wrote: post #26

Earlier quoted context omitted.

For me this is DOA unless it can be run locally without having to call into some hosted API in the cloud. I took a gander at the source I'll wait till/if there's ever a 100% local version of this. But good luck all the same!

I guess this is your lucky day, because you can already run it locally! Take a look at the "Local Testing" section in the README of dispatch-py https://github.com/stealthrocket/dispatch-py?tab=readme-ov-f... TLDR: python -m dispatch.test http://127.0.0.1:8000 This is just a mock implementation but it lets you test your code and experiment with the solution.

Still production means sending my data to a hosted cloud black box? I can’t host this myself?

Re: Distributed coroutines with a native Python extension and Dispatch

#29

Post by achille-roussel »

gigatexal wrote: post #28
achille-roussel wrote: post #27

Earlier quoted context omitted.

I guess this is your lucky day, because you can already run it locally! Take a look at the "Local Testing" section in the README of dispatch-py https://github.com/stealthrocket/dispatch-py?tab=readme-ov-f... TLDR: python -m dispatch.test http://127.0.0.1:8000 This is just a mock implementation but it lets you test your code and experiment with the solution.

Still production means sending my data to a hosted cloud black box? I can’t host this myself?

Yes, today the deployment model for production is to connect to a cloud service for the scheduling. You still run the code yourself, but the SDK needs to connect to the backend.

Re: Distributed coroutines with a native Python extension and Dispatch

#30

Post by traverseda »

achille-roussel wrote: post #8
faizshah wrote: post #5

I feel like temporal’s approach of replaying the computation is probably better than trying to serialize the running computation. Serializing the running coroutine gets ugly as is shown here with things like file handles and making pickle a central part of your compute platform is a little scary from an AppSec pov. That being said, I like the idea and the blog post is wonderfully written.

Dispatch takes a different approach with different trade-offs, but you're right that capturing the local program scope brings interesting challenges. Serializing file handles doesn't work, but in our experience, programs rarely run into constructs where this becomes a problem, and when it happens, there are mitigation measures that are usually easy to implement (small restructure of the program, capturing resource me…

What about combining it with rpyc for network transparent object proxies? For resources that can't be picked.
Post reply on HN