Queues Aren't the Right Abstraction • Hacker News Board

Re: Queues Aren't the Right Abstraction

#11

Post by zer00eyz » Thu, Mar 28, 2024, 5:20 PM UTC

goodoldneon wrote: post #4

zer00eyz wrote: post #2

This feels like it turns into an ESB? We went from ESB's because the were opaque. And this seems like it's very opaque. All that work that you put into queues... its ugly but it's transparent. You can rationalize it, you're not abstracting things away into magic.

But you also spend a lot of cycles building and maintaining the ancillary features that make queues powerful. Early- to mid-stage companies especially need to focus on business logic and less on reinventing wheels

>>> Early- to mid-stage companies especially

If your early to mid-stage company needs queues to depth then what is in them is vital to your business.

You better know what is going on with mission critical data at every step of its journey. Or the data isnt that important and you should write something lighter than this because it can be lossy...

Re: Queues Aren't the Right Abstraction

#12

Post by greenpinia » Thu, Mar 28, 2024, 5:34 PM UTC

If you’ve had a conversation with your team about using durable workflows instead of queues, how did you get a buy-in?

Re: Queues Aren't the Right Abstraction

#13

Post by lijok » Thu, Mar 28, 2024, 5:41 PM UTC

lelanthran wrote: post #5

> You need queues for three core reasons: > They offer reliability through guaranteed delivery, persistence, and dead letter queues, so developers know they aren't sending workloads into a black hole. I disagree with this reason to use queues. If this is the only reason for using SQS or RabbitMQ or similar, perhaps the application is over-engineered. If you want reliability, and that alone, use a transaction-based sy…

Could you elaborate on how a transaction-based system enables durable execution?

Re: Queues Aren't the Right Abstraction

#14

Post by moribvndvs » Thu, Mar 28, 2024, 6:03 PM UTC

We keep reaching for them because they work fine most of the time. This article would have been less click-bait and possibly more persuasive if instead of “QuEuEs R oVeR” you would have simply just explored some advanced scenarios that are tricky to solve and prescribed how your product makes it worth another line item on our monthly bill.

Re: Queues Aren't the Right Abstraction

#15

Post by claytonjy » Thu, Mar 28, 2024, 6:23 PM UTC

greenpinia wrote: post #12

If you’ve had a conversation with your team about using durable workflows instead of queues, how did you get a buy-in?

have them read blog posts, docs, and client code from Temporal

Re: Queues Aren't the Right Abstraction

#16

Post by goodoldneon » Thu, Mar 28, 2024, 7:24 PM UTC

zer00eyz wrote: post #11

goodoldneon wrote: post #4

Earlier quoted context omitted.

But you also spend a lot of cycles building and maintaining the ancillary features that make queues powerful. Early- to mid-stage companies especially need to focus on business logic and less on reinventing wheels

>>> Early- to mid-stage companies especially If your early to mid-stage company needs queues to depth then what is in them is vital to your business. You better know what is going on with mission critical data at every step of its journey. Or the data isnt that important and you should write something lighter than this because it can be lossy...

Queues and workflow engines aren’t the right solution for everything, but they work well for a lot of stuff. Like a signup flow that integrates with multiple third-parties, or a drip campaign, or notifications.

Those are things many small, all developer teams need. They don’t want to hire an infra-minded person just for queues and workflows

Re: Queues Aren't the Right Abstraction

#17

Post by 1letterunixname » Thu, Mar 28, 2024, 7:47 PM UTC

This press release content marketing appears hung up on some mythical perfect ESB system containing kitchen sink cross-cutting concerns.

There are many tools in the toolbox for backend infrastructure: nosqls (memcache/redis/keydb), dlms (zk), kafka, rabbitmq, ejabberd, 0mq, nng. Some scale better than others, and some are more atomic or durable than others. OLTP and infrastructure orchestration will have different needs. Sometimes, cross-cutting concerns can be added by gating the sender, receiver, or both with "controller"-like middleware proxies or modifications.

"Use good judgement" is the prime directive.

Re: Queues Aren't the Right Abstraction

#18

Post by jesse11 » Thu, Mar 28, 2024, 11:34 PM UTC

Having built my own queue-based data processing system and worked through a lot of pain in the past, I will say that I'm a huge fan of Inngest and what they are doing.

As for the article, I think the main point being driven here is:

"building a system with queues requires much more than just the queue itself"

I would imagine almost anyone who has built a production grade queue-based message processing system would agree.

For me, I would say that for the majority of software being developed, the investment to build all of that yourself just doesn't make sense. Obviously there will be exceptions, but Inngest gives you incredible power at a very simple layer of abstraction.