151–159 of 159 posts
how is this compared to hatchet?
Happy to answer more questions beyond this!
I do love Postgres and DBOS. I also recently started experimenting with https://github.com/earendil-works/absurd which is also Postgres and even simpler than DBOS. Their comparison is a great read: https://earendil-works.github.io/absurd/comparison/ But for operational reasons I've started using sqlite for durable workflows instead. Porting the database concepts from either DBOS or absurd PG to SQLite is remarkably e…
A dedicated external orchestrator decouples application servers' API and implementation from that of the orchestrator. This comes with some advantages. It becomes easier and more natural for the application servers to organize their APIs and implementation around a stable set of domain areas (instead of operating along two different layers of abstraction -- the workflow orchestration plane, and the underlying application domain logic). It also becomes easier to understand and update workflow logic when it's managed separately as a first-class citizen as opposed it being fragmented and diffused across multiple application servers. In my experience, these advantages tend to matter more in highly distributed system involving multiple semi-independent teams owning multiple application servers and data-stores. After a certain point in terms of the size and complexity of the distributed system, the aggregate cost of handling orchestration and checkpointing workflows often starts to provide stronger justification for having dedicated centralized orchestrator.
So while I agree that letting go of the central orchestrator (and letting the application servers and data-store do that work on their own) can sometimes be the pragmatic/preferred option, I'd argue that the fit is context dependent and there doesn't seem to be a one-size-fits-all solution available.
Earlier quoted context omitted.
Did I say revolutionary?
revolutionary, revelatory whatever. but no your comment was just empty platitudes
> Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay/debugging semantics, workflow migration, fanout/fanin, long timers, audit trails, and operator tooling, the “just use a database” story becomes “build a poor copy of a workflow engine plus a bunch of workers.” pretty quick.
Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…
DBOS is much less complexity compared to Temporal. That’s the benefit. Main tradeoff is lower performance. Or at least, you’re going to be limited to what you can push through Postgres. If that’s sufficient for your needs DBOS is great.
Tens of thousands of workflows per second
Earlier quoted context omitted.
I was not clear; I did not mean not 200 a day, it's 10s of thousands of concurrently running workflows, sometimes into the hundreds of thousands, each with 200 events. We run many hundreds of thousands of these a day. Temporal was a bad fit for us, and we regret it deeply.
Ah. So multiple billion actions per month, and probably multiple million dollars per year on their cloud, if they can even support that load (plus, the vendor lock in and etc). Makes sense.
Given the open source nature of the product, I'm not sure what vendor lock-in you mean. Migrating off cloud to your own infra again is doable. https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-...
Earlier quoted context omitted.
Ah. So multiple billion actions per month, and probably multiple million dollars per year on their cloud, if they can even support that load (plus, the vendor lock in and etc). Makes sense.
I'm reasonably sure their cloud product can handle that. you might be in the scale where they want to hold your hand in the transition though. Given the open source nature of the product, I'm not sure what vendor lock-in you mean. Migrating off cloud to your own infra again is doable. https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-...
As our CEO and co-founder Samar Abbas stated in the keynote at this year's Replay conference (https://youtu.be/BxEB7Y2U9oU?t=532), Temporal Cloud is currently handling 35 billion workflows per day. If you assume that each Workflow averages 10 actions, that translates to more than 10 trillion actions per month.
Later in that keynote, a VP from OpenAI talks about the scale at which they are using Temporal Cloud. That'll give you a sense of the volume for a single customer (obviously a very large one).