Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Temporal Python – A durable, distributed asyncio event loop (2023)

temporal.io

Post reply on HN
Print view View original

51–56 of 56 posts

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#51

Post by 7bit »

pm wrote: post #48
7bit wrote: post #22

Earlier quoted context omitted.

Ohh, that's great to hear! I do like ADF, but the Python worker is full of bugs and weird behaviour and tickets stay open for month without progress. I will definitely check that out!

Working on a freelance job 3 years ago, I got sucked down a rabbit-hole for months trying to get Azure Durable Functions to work. Too many bugs, no visibility into its workings and the worker would always grind to a halt. That job did not go well. Avoid ADF (let alone Azure) until you've got some innovation tokens to spend, or go with another provider.

It got much better. The internals are explained well enough, although there is room for improvement. The bugs can be worked around. It's super annoying, but are the lesser evil when compared to what ADF brings to the table. However, I'm looking into Temporal to see, if that's the better app.

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#52

Post by wojcikstefan »

ub-volta-toss wrote: post #46

Temporal is really neat but I think its marketed at too many use cases. After a year of high-scale Temporal work, I found it was only good for low-scale work. The onboarding and learning curve were insanely difficult and complex. Ultimately it doesn't scale as well as you think. The temporal team invented their own database to get around this limitation.

Can you tell us more about your scaling issues with Temporal?

I haven't yet used it in production, but I would've expected that a system which evolved out of Uber's Cadence [0] (and which I believe is used at Uber extensively) would've scaled very well.

[0]: https://stackoverflow.com/a/61281435/1579058

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#53

Post by ub-volta-toss »

wojcikstefan wrote: post #52
ub-volta-toss wrote: post #46

Temporal is really neat but I think its marketed at too many use cases. After a year of high-scale Temporal work, I found it was only good for low-scale work. The onboarding and learning curve were insanely difficult and complex. Ultimately it doesn't scale as well as you think. The temporal team invented their own database to get around this limitation.

Can you tell us more about your scaling issues with Temporal? I haven't yet used it in production, but I would've expected that a system which evolved out of Uber's Cadence [0] (and which I believe is used at Uber extensively) would've scaled very well. [0]: https://stackoverflow.com/a/61281435/1579058

I'm not sure how Uber does it, but it might be because they're using Cadence instead.

The Temporal team has acknowledged that Cassandra-backed Temporal hits scaling limits pretty fast.

The limitations aren't a clean "X actions/sec", they're sneakier. Because you can run X/sec for days and then the memory on the history service will spike, or any tiny slowdown in the DB will cause looping degradation. There are nasty feedback loops hidden in Temporal that turn small problems into very very large problems.

I think the core problem with Temporal is the way its sharded. This affects history service and its caches. If anything tells them to reload or restart, or that any of the nodes are unreachable, you get a retry storm on the DB.

In addition to these issues, Temporal can create feedback loops within itself. I've seen cases where it would not return to health, even with 0 workers requesting work for 10s of minutes.

We could have kept using and scaling Temporal, but it required 10-30x the resources of building something else. And it was scary to administer. You really need an entire team. You can't have somebody who isn't a dedicated engineer take on-call for it.

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#54

Post by ub-volta-toss »

NoThisIsMe wrote: post #49
ub-volta-toss wrote: post #46

Temporal is really neat but I think its marketed at too many use cases. After a year of high-scale Temporal work, I found it was only good for low-scale work. The onboarding and learning curve were insanely difficult and complex. Ultimately it doesn't scale as well as you think. The temporal team invented their own database to get around this limitation.

Invented their own database? They use Cassandra IIRC

Nope. They hit the scale limit with Cassandra and now have an in-house storage layer

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#55

Post by ub-volta-toss »

claytonjy wrote: post #47
ub-volta-toss wrote: post #46

Temporal is really neat but I think its marketed at too many use cases. After a year of high-scale Temporal work, I found it was only good for low-scale work. The onboarding and learning curve were insanely difficult and complex. Ultimately it doesn't scale as well as you think. The temporal team invented their own database to get around this limitation.

Would love to hear more about the scale issues you saw. How many workflows or actions was too many? which components started breaking down, what were their failure modes?

See above. Its not so straightforward. You need enough headroom on each component that a negative feedback loop can start, eat resources, and have enough time and resources to calm itself before hitting some limit or degrading itself further

Re: Temporal Python – A durable, distributed asyncio event loop (2023)

#56

Post by claytonjy »

ub-volta-toss wrote: post #53
wojcikstefan wrote: post #52

Earlier quoted context omitted.

Can you tell us more about your scaling issues with Temporal? I haven't yet used it in production, but I would've expected that a system which evolved out of Uber's Cadence [0] (and which I believe is used at Uber extensively) would've scaled very well. [0]: https://stackoverflow.com/a/61281435/1579058

I'm not sure how Uber does it, but it might be because they're using Cadence instead. The Temporal team has acknowledged that Cassandra-backed Temporal hits scaling limits pretty fast. The limitations aren't a clean "X actions/sec", they're sneakier. Because you can run X/sec for days and then the memory on the history service will spike, or any tiny slowdown in the DB will cause looping degradation. There are nasty…

What did you move to instead?
Post reply on HN