11–20 of 45 posts
Regarding the "Durable Queueing Tradeoffs", doesn't Kafka prove you can be both durable and highly performant?
Kafka is a wonderful technology that punts on the most difficult part of distributed stream processing and makes it the consumer's problem.
While this does talk about durable queues , this post really hinges on a pivot to evangelizing durable workflows . It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durabl…
The reason none of the others were mentioned is because they all work very differently than DBOS. All of those others require an external durability coordinator, and require you to rewrite your application to work around how they operate.
DBOS is a library that does its durability work in process and uses the application database to store the durability state. This means the latency is much smaller, and the reliability is much higher because there aren't extra moving parts in the critical path that can go down.
Here is a page about this difference: https://docs.dbos.dev/architecture
While this does talk about durable queues , this post really hinges on a pivot to evangelizing durable workflows . It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durabl…
I also now have the dreadful notion of debugging a non-deterministic deadlock or race condition in a workflow that takes a week to run!
While this does talk about durable queues , this post really hinges on a pivot to evangelizing durable workflows . It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durabl…
Instead of forcing you into a custom async runtime, DBOS lets you keep writing normal functions (this is an example in Python):
@DBOS.workflow()
def do_thing(foo):
return bar
# You can still call the workflow function like this:
result = do_thing(fooInput)
On top of that, DBOS also supports running workflows asynchronously or through queues, so you can start with a simple function call and later scale out to async/queued execution without changing your code. That's what the article was leading into.
While this does talk about durable queues , this post really hinges on a pivot to evangelizing durable workflows . It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durabl…
Thanks. I found that very informative! I also now have the dreadful notion of debugging a non-deterministic deadlock or race condition in a workflow that takes a week to run!
Also, check out the sibling comment for more information about durability.
There are good things about this article. It seems like a nice intro for people who have never worked on this type of system. And I'm always happy to celebrate someone's win. But I feel like it was missing an overview of what actually makes distributed queues difficult (the distributed part) and why you probably don't need them if you can survive without them.
Reading it I imagine it's roughly because they started with the problem of "we have to async writes to postgres for scale" and then solved it with "we synchronously write checkpoints with enough performance and guarantees to postgres to solve scale". The middle bit was likely quite hard.
There are good things about this article. It seems like a nice intro for people who have never worked on this type of system. And I'm always happy to celebrate someone's win. But I feel like it was missing an overview of what actually makes distributed queues difficult (the distributed part) and why you probably don't need them if you can survive without them.
> How I solved a distributed queue problem after 15 years
Well… how? The post is a nice description of durable queues, but it never explicitly says they’re a solution to a distributed queue problem, nor does it specifically define such a problem.
Is “durable queue” a brand name of a DBOS feature? Because the post doesn’t even say “and here’s how you can use DBOS for durable queues,” nor does it compare it to Kafka or any other “durable queue” solution that’s emerged in the fifteen years since the author used RabbitMQ… (btw, isn’t RMQ a durable queue…?)
Regarding the "Durable Queueing Tradeoffs", doesn't Kafka prove you can be both durable and highly performant?
Kafka is great for streaming use cases, but the big advantage of Postgres-backed queues is that they can integrate with durable workflows, providing durability guarantees for larger programs. For example, a workflow can enqueue many tasks, then wait for them to complete, with fault-tolerance guarantees both for the individual tasks and the larger workflow.
There’s nothing inherently different about the durability of Postgres that makes it better than Kafka for implementing durable workflows. There are many reasons it’s a better choice for building a system like DBOS to implement durable workflows – ranging from ergonomics to ecosystem compatibility. But in theory you could build the same solution on Kafka, and if the company were co-founded by the Kafka creators rather than Michael Stonebraker, maybe they would have chosen that.
While this does talk about durable queues , this post really hinges on a pivot to evangelizing durable workflows . It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durabl…
More recently discussed are OCaml's effect system, or Flix programming language.