[dead]
Building durable workflows on Postgres
141–150 of 159 posts
Re: Building durable workflows on Postgres
#142Continuously amazed by what you can do with few tools, as long as Postgres is a part of your toolkit. I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely. You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.
Just do a reservation on the record with the actor ID.
Let's say the actor then crashes, how do you recon and have it pick up again?
Re: Building durable workflows on Postgres
#143This feels like the sort of architecture that starts clean and then gradually grows most of the things a workflow-native system already has. I've seen systems like this, seen companies that are built out of this idea, and built small systems like this over time. Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay…
Comments like this by people who know exactly what they are talking about are why I love Hackernews
Re: Building durable workflows on Postgres
#144Rails has several database-backed job backends, but the convention is always to make jobs do one thing, and ideally be very short-lived. This makes building workflows a bit contrived: we end up enqueuing the second job on the last line of the first one, enqueuing the third one on the last line of the second one, etc. The job backend treats these as independent jobs rather than showing them as a connected workflow, and you have to read through a bunch of job classes to wrap your head around the workflow at even a high level
Rails recently introduced a 'continuable' concept, allowing you to checkpoint and resume steps within a job, but it still feels like the convention is too keep jobs with a single responsibility, so it feels odd to use them for true workflows.
Has anyone else experienced this or found a solution to it?
Re: Building durable workflows on Postgres
#145As someone who uses dbos.dev, restate.dev, cf workflows here is a snippet from our Agents.md: Restate.dev: for payment integrations on northflank since its faster than cf workflows, independent of cf and its downtime and self-hostable vendor-lock-in free, Cloudflare workflows: for non critical stuff like csv/pdf report generations since it's very cheap. DBOS.dev: for workflows that need atomic messaging tied to a pos…
how are you handling schema updates? do you migrate jobs or handle worker deployments in a specific way?
Re: Building durable workflows on Postgres
#146I am not convinced that using a special software for "durable workflows" is necessary. If one has a stateful message queue or job task queue, e.g. RabbitMQ or Celery, one can use it. Irrespective, many jobs can be made idempotent. The most that you ought to residually need is a column in an existing table of your own database which keeps track of what remains to be done. Given the above, it would seem that durable wo…
I've talked to dozens of engineers who built their home grown "durable" stack. Most of them eventually moved on to buying vs building, when their system actually scaled. It's just not a side-hustle to build a foundational reliability layer.
That argument comes down to the scalability of RabbitMQ or one's database, both of which can scale fairly well, but require tuning. In the absolute worst case, one would have to use a distributed cloud database, e.g. AWS Aurora or AWS DynamoDB, otherwise a self-hosted one, e.g. TiDb or YugabyteDb, but far less than 1% of users would even need anything like it.
In the pre-AI era, the argument of using a third party tool or service even had some weight, but today, AI can even do much of the heavy work when pointed in the right direction wrt using the aforementioned. For the majority of users, a SQLite database will do the job.
Re: Building durable workflows on Postgres
#147Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…
[flagged]
Re: Building durable workflows on Postgres
#148This feels like the sort of architecture that starts clean and then gradually grows most of the things a workflow-native system already has. I've seen systems like this, seen companies that are built out of this idea, and built small systems like this over time. Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay…
Ridiculously good analysis! HN is a national treasure because of posts like this.
What was so revolutionary to you in their post to cause you to describe it as a “ridiculously good analysis”?
Re: Building durable workflows on Postgres
#149Re: Building durable workflows on Postgres
#150[flagged]