Temporal Python – A durable, distributed asyncio event loop (2023)

I found out about Temporal 2+y ago now and were early adopters of their cloud. It's a bit of a paradigm shift when you start using it, but it is amazing at solving some types of problems in a very simple manner. There are some trade-offs, there always are, for us that was migrating long lived workflows. But the resulting simplicity and maintainability of the code has been great. One thing that was hard with Temporal was to "sell it" to business leaders because it's not a turn key solution, it's more like a piece of infra for engineers to build on top of. Kind of a higher level database-queue-workflow engine thing that simplifies work for engineers.

In short we were working on automating things like onboarding a new employee, which involves creating accounts for their saas apps, buying and shipping their device, email confirmations, satisfactions surveys etc. So a workflow could last up to 3 months with some fully automated systems, and some that required integrating with people (listening to jira event to trigger things, etc).

The error handling was the thing that sold me on Temporal, because things can break just about anywhere in unpredictable ways (not just code, can be process, employee quits during the onboarding, customer is out of licenses etc), so we need everything to be robust and be fixable by a person. With homegrown queue based systems or with BPEL it can be hard handling these situations (what if you need to roll back 3 steps?). With code you can use exceptions, write unit tests etc. We use the typescript sdk, promises made it very intuitive to code even some otherwise complicated scenarios (say event listeners etc).

We migrated from an in-house redis queuing system.

Temporal has its own way of doing things; there's rules about what you can and cant do in workflows, what has to live in activities, etc. Its generally quite easy to adapt existing code work with it. We use typescript.

The worst part for us has been error/anomaly handling. Workflows can sometimes hit a state where the status reads in progress and errors aren't reported anywhere except buried in the event log; which surfaces great in the UI but we still haven't figured out how to programmatically respond to this condition.

A good example is: we use a home-grown version of this [1] to proxy large payloads to S3. However, if those payloads get REALLY large, they can take some time to upload and download; and if that "some time" is longer than 5 seconds, the control plane will believe that the worker has died, it won't reschedule, and the workflow just sits in In Progress. There's always a beautiful error on the temporal dashboard, and we can manually terminate/retry, but the world just seems to die when this happens and we can't do error-level cleanup stuff like alert the user that the thing they were doing didn't finish.

Temporal is also challenging to get support for. Its new, open source, we don't pay for temporal cloud, and there's not a ton of resources or people using it. The documentation is quite bad (if you like 500,000 word pages, codegen'd library sites with no comments, and one example for each feature, you'll like their documentation). Given we run our own temporal cluster, we've also had pretty large challenges in the self-hosting world. We work through them, usually after deep-diving into the temporal server code itself, but there's startlingly little documentation on self-hosting, and even less community support.

Overall, we don't regret adopting it, but if we had a time machine we wouldn't do it again. I feel it makes a series of sacrifices in order to create a system that has extremely high standards for processing, like financial/bank/healthcare level stuff. But, not only are we not building that, but the system has never behaved in a way which makes me think I'd even want to use it if I worked in those industries. Obviously I feel like I'm the one in the wrong here, and I'm sure its just a matter of "we screwed up something somewhere", but that leads back to: bad documentation, no way to get professional support without being on their cloud, and a lack of community support.

[1] https://github.com/DataDog/temporal-large-payload-codec

The OP seems targeted at devs who are already quite familiar with Temporal and are interested in using the new Python exposure.

FWIW, as someone who has never previously encountered Temporal, and has only a vague sense of the specific problem set it's trying to tackle and architectural approach it has taken, I find the post to be fairly impenetrable.

I'd love to read a proper introduction to Temporal by way of Python (and probably also by comparison to Celery).

Distributing the work across machines essentially comes for free with the ability to replay workflows from any step. You could run all of your worker processes on a single machine if it had enough capacity, but resuming a workflow on a different machine is transparent to your workloads assuming there's no local state.

> Workflow versioning is one thing we suck at still

well its not entirely your fault :) what practices have you adopted now that you have some experience with it?

lower down OP mentions that they got the link from a HN discussion on asyncio 2 days ago https://news.ycombinator.com/item?id=40287354 . i guess the upvotes are today's lucky 10,000 learning about it for the first time.