Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Temporal Python – A durable, distributed asyncio event loop (2023)

temporal.io

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#11

Post by blegr »

uniqueuid wrote: post #6
bmitc wrote: post #3

As someone familiar with asyncio, I don't understand what this is or what it's for. What's an activity, workflow, or worker? > See the asyncio.sleep in there? That’s no normal local-process sleep; that’s a durable timer backed by Temporal. That's the normal asyncio.sleep. What does backed by Temporal mean? Reading further, it appears that Temporal is replacing the default asyncio event loop. I don't understand why ev…

The thing with event loops in python is that they are not a single, all-governing scheduler (as e.g. in the BEAM). ev loops instead are a mid-layer concept that sits below other infrastructure such as threads and processes. And (perhaps somewhat frustratingly) it is not too uncommon to have multiple ev loops in parallel. See for example the proxy.py project, which offers to run one async loop per process for a speedu…

Good design dictates that you start one loop and build the whole program around it, no? The docs for asyncio.run say as much.

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#12

Post by ikari_pl »

KaiserPro wrote: post #10

Isn't this just threads but with more surprise gotos?

no, the whole point of temporal is to distribute work across machines, but without worrying too much on the orchestration.

workflows and activities are called remotely, and you can have an autoscaled worker pool handling these calls.

you can retry any unit easily on failure and specify the non retryable errors. What it requires in exchange is full determinism - the same input should produce the same activities in the same order, as a good starting point.

src: I'm a user since over a year ago.

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#14

Post by kodablah »

avi_vallarapu wrote: post #7

Relying on external APIs or databases within activities might lead to variability in workflow execution. Also, on handling HTTP errors in activities by raising an "ApplicationError" based on the status code, might simplifies error handling but might need to see how it accounts for more complex scenarios where errors are transient or where a retry could be successful even for some client errors like rate limiting or t…

> Relying on external APIs or databases within activities might lead to variability in workflow execution.

This is why they are activities. Their results are stored in history, the workflow remains deterministic.

> might need to see how it accounts for more complex scenarios where errors are transient or where a retry could be successful even for some client errors like rate limiting or temporary unavailability etc.

Temporal allows you to specify whether an error is retryable or not.

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#16

Post by kodablah »

benakh wrote: post #15

Anyone migrated from celery, with / without regrets?

Many Temporal users used Celery in the past. There was a popular blog post a while back about issues with celery: https://steve.dignam.xyz/2023/05/20/many-problems-with-celer.... Here's a brief heading-by-heading listing of how Temporal addresses those issues: https://community.temporal.io/t/suggestion-for-blog-post-abo....

(disclaimer, I'm the author of the post)

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#17

Post by kodablah »

bmitc wrote: post #3

As someone familiar with asyncio, I don't understand what this is or what it's for. What's an activity, workflow, or worker? > See the asyncio.sleep in there? That’s no normal local-process sleep; that’s a durable timer backed by Temporal. That's the normal asyncio.sleep. What does backed by Temporal mean? Reading further, it appears that Temporal is replacing the default asyncio event loop. I don't understand why ev…

To add to other responses here, Temporal doesn't take over the default event loop in general (and users still use it for clients and activities and such). Temporal workflows must be deterministic and durable which means they are guaranteed to run and are resumable on other machines. Therefore Temporal workflows specifically operate on a custom event loop implementation. It doesn't affect anything outside the workflow.

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#18

Post by robertlagrant »

bmitc wrote: post #3

As someone familiar with asyncio, I don't understand what this is or what it's for. What's an activity, workflow, or worker? > See the asyncio.sleep in there? That’s no normal local-process sleep; that’s a durable timer backed by Temporal. That's the normal asyncio.sleep. What does backed by Temporal mean? Reading further, it appears that Temporal is replacing the default asyncio event loop. I don't understand why ev…

It's like that old Joel Spolsky article[0] says:

> you only have to get one supergenius to write the hard code to run map and reduce on a global massively parallel array of computers, and all the old code that used to work fine when you just ran a loop still works only it’s a zillion times faster which means it can be used to tackle huge problems in an instant

If you can replace the thing that people use with a distributed version, then that can make it easy to write distributed code.

[0] https://www.joelonsoftware.com/2006/08/01/can-your-programmi...

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#19

Post by kodablah »

blegr wrote: post #11
uniqueuid wrote: post #6

Earlier quoted context omitted.

The thing with event loops in python is that they are not a single, all-governing scheduler (as e.g. in the BEAM). ev loops instead are a mid-layer concept that sits below other infrastructure such as threads and processes. And (perhaps somewhat frustratingly) it is not too uncommon to have multiple ev loops in parallel. See for example the proxy.py project, which offers to run one async loop per process for a speedu…

Good design dictates that you start one loop and build the whole program around it, no? The docs for asyncio.run say as much.

Yes, that is good design and the event loop should basically be shared process-wide (asyncio objects are usually not thread safe and cannot be shared across event loops). Temporal only does custom event loops in isolated workflows.
Post reply on HN