Queues Aren't the Right Abstraction
inngest.com
Queues Aren't the Right Abstraction
1–10 of 18 posts
Re: Queues Aren't the Right Abstraction
#2This feels like it turns into an ESB?
We went from ESB's because the were opaque. And this seems like it's very opaque.
All that work that you put into queues... its ugly but it's transparent. You can rationalize it, you're not abstracting things away into magic.
Re: Queues Aren't the Right Abstraction
#3That was a lot of reading to discover they’ve built a workflow engine. They don’t want to call it a workflow engine because there’s a million of those. So they can up with a new name for a workflow engine.
Re: Queues Aren't the Right Abstraction
#4This feels like it turns into an ESB? We went from ESB's because the were opaque. And this seems like it's very opaque. All that work that you put into queues... its ugly but it's transparent. You can rationalize it, you're not abstracting things away into magic.
But you also spend a lot of cycles building and maintaining the ancillary features that make queues powerful. Early- to mid-stage companies especially need to focus on business logic and less on reinventing wheels
Re: Queues Aren't the Right Abstraction
#5> You need queues for three core reasons:
> They offer reliability through guaranteed delivery, persistence, and dead letter queues, so developers know they aren't sending workloads into a black hole.
I disagree with this reason to use queues. If this is the only reason for using SQS or RabbitMQ or similar, perhaps the application is over-engineered.
If you want reliability, and that alone, use a transaction-based system.
Re: Queues Aren't the Right Abstraction
#6That was a lot of reading to discover they’ve built a workflow engine. They don’t want to call it a workflow engine because there’s a million of those. So they can up with a new name for a workflow engine.
Yup, looks like a hopeful competitor to Temporal but too scared to name them directly.
Re: Queues Aren't the Right Abstraction
#7Spot on. This is what Durable Functions does on Azure and it's brilliant for implementing complicated business processes and handling multiple events in one flow, in a resilient way where the logic is easy to follow.
One catch is that you're going to have to version your code if your workflows/orchestrations run for days, or if there's no windows without running workflows. And there's no built-in support for this, so expect to duplicate your entire workflow for new versions, so the old one can run to the end with the old code.
Re: Queues Aren't the Right Abstraction
#8Get to the point: Temporal is lit and you're building a competitor to it
Re: Queues Aren't the Right Abstraction
#9> You need queues for three core reasons: > They offer reliability through guaranteed delivery, persistence, and dead letter queues, so developers know they aren't sending workloads into a black hole. I disagree with this reason to use queues. If this is the only reason for using SQS or RabbitMQ or similar, perhaps the application is over-engineered. If you want reliability, and that alone, use a transaction-based sy…
How are you going to do that over network boundaries?
Especially the ones you might have less control over.
Re: Queues Aren't the Right Abstraction
#10Inngest is very cool, and if you are interested in the topic but absolutely want to self-host, 2 other high-performance durable execution engines that enable all the above and that you can self-host and are open-source and mature would be temporal and windmill. Both can use postgresql as the queue (temporal can use cassandra too), the rest can be done leveraging the transactional properties of postgresql, such as atomic counters for the concurrency keys: https://github.com/windmill-labs/windmill/blob/main/backend/... or re-queuing jobs that haven't progressed when they should have (most likely, worker crashed).
There are very cool things you can do today without too much complexity with the primitives that modern database are capable of. Should one rebuild it for their startups, no, but if you were to extract the very core of the durable execution engine of windmill for instance, it would actually be surprisingly reasonable given that postgresql does the heavy lifting. I strongly believe the benefits of our platforms mostly come from the overall virtue to be standardized, opinionated and working out-of-the-box in a way that make everything fit together rather than the overall engineering complexity of it.