Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

341–350 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#341

Post by n_ary »

IMHO, message queues were all the hype at the time because cloud was picking up steam and people discovered how to decouple and handle massive loads.

For the blog posts, most were garbage(and still are) if my memory serves right. I recall reading a lot of blog posts and all of those were basically a different derivative of same “quick-start tutorial” that you would find on any decently documented piece of software. Once you delve into the real trenches, the blog posts start showing their limits immediately and the shallowness of the depth.

That all being said, message queues are very crucial part of most complex systems these days, same as your typical tools(containers, git, your language of choice etc.), it has moved onto mature and boring.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#342

Post by synthc »

People discovered that using messaging can add tons of overhead, and async messaging architectures are harder to manage and monitor.

Serialization and going over the network are an order of magnitude slower and error prone than good ol' function calls.

I've seen too many systems that spent more time on marshalling from and to json and passing messages around than actual processing.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#343

Post by everforward »

ndthrowaway wrote: post #212
sofixa wrote: post #140

Earlier quoted context omitted.

> And then there is Helm Right, who doesn't want to template hundreds of lines of code in a language that uses whitespace for logic and was never made neither for templating nor complex long documents(YAML)? What could possibly go wrong ("error missing xxx at line 728, but it might be a problem elsewhere").

"in a language that uses whitespace for logic" This argument kind of died when python became one of the most popular programming languages used.

It’s not completely that white space is bad, but in particular that white space is very difficult to template relative to something character-delimited like JSON or s expressions.

In JSON or s expressions, the various levels of objects can be built independently because they don’t rely on each other. In YAML, each block depends on its parent to know how far it should be indented, which is a huge pain in the ass when templating.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#344

Post by Noneofya »

pjlegato wrote: post #304
bamboozled wrote: post #294

Earlier quoted context omitted.

Depending on your product, this could mean tens of thousands to millions of dollars worth of revenue loss. I don't really see how we've gone backwards here. You could just distribute your workloads using...a queue, and not have this problem, or have to pay for and pay to maintain backup equipment etc.

If your product going down for an hour will lead to the loss of millions of dollars, then you should absolutely be investing a lot of money in expensive distributed and redundant solutions. That's appropriate in that case. The point here is that 99% of companies are not in that scenario, so they should not emulate the very expensive distributed architectures used by Google and a few other companies that ARE in that s…

> The point here is that 99% of companies are not in that scenario

I‘d argue that is wrong for any decently sized ecommerce platform or production facility. Maybe not millions per hour, but enough to warrant redundancy. There’s many revnue and also redundancy levels between Google and your mom and pop restaurant menu.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#345

Post by Noneofya »

sturmdev wrote: post #308
bamboozled wrote: post #294

Earlier quoted context omitted.

Depending on your product, this could mean tens of thousands to millions of dollars worth of revenue loss. I don't really see how we've gone backwards here. You could just distribute your workloads using...a queue, and not have this problem, or have to pay for and pay to maintain backup equipment etc.

From the original post: “Your business is not Google and will never be Google” From the post directly above: “Most businesses…” The thread above is specifically discussing business which won’t lose a significant amount of money if they go down for a few minutes. They also postulate that most businesses fall into this category, which I’m inclined to agree with.

Hours, not minutes. That is relevant for most businesses.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#346

Post by memhole »

appplication wrote: post #273
RangerScience wrote: post #108

Earlier quoted context omitted.

Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things…

I think there are some questions to ask that can help drive your system design here. Does each node in the DAG represent an event at which some complex automated logic would happen? If so, then I think the above would be recommended, since most of your logic isn’t the DAG itself, and the DAG is just the means of contextually triggering it. However, if each node is more of a data check/wait (e.g. we’re on this step un…

Do you avoid things like task sensors? Based off what you described it sounds like an anti pattern if you’re using them.

Great description of good orchestration design. Airflow is fairly open ended in how you can construct dags, leading to some interesting results.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#347

Post by EVa5I7bHFq9mnYK »

coffeecloud wrote: post #321
EVa5I7bHFq9mnYK wrote: post #198

Earlier quoted context omitted.

Because JS is single threaded, and an average programmer today is too dumb to learn any language beyond JS, you must build everything into "microservices".

You sound like a joy to work with

Im retired so can tell the truth now)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#348

Post by arantius »

benterix wrote: post #178
keredson wrote: post #122

Earlier quoted context omitted.

and mandates that virtually all new projects not directly use borg/kubernetes.

That's interesting. What's the rationale behind that?

I left Google in 2022, but I remember seeing the beginning of this. The system that got built up to do all the things Google wanted to do -- previously "simple" things, orchestrated on raw hosting (Borg) -- got too big and complex. (Partially because it was too flexible, so too many projects did too many unique things, and running them grew difficult to match.) So they started adding abstraction layers, to make it "simpler". (Which as always means the common/easy case got easier, and the rare/difficult case got harder. The team I was on at the time couldn't use the new thing for (most) of what we did, because our needs were incompatible. But we did enough other compatible things that I started learning about it/using it.)

(Personally I'm sure it's doomed to failure. But partially because most all technology operates in cycles/like a pendulum. The abstraction layers will grow too onerous and limiting, and the solution will be to dive closer to the metal again.)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#349

Post by appplication »

memhole wrote: post #346
appplication wrote: post #273

Earlier quoted context omitted.

I think there are some questions to ask that can help drive your system design here. Does each node in the DAG represent an event at which some complex automated logic would happen? If so, then I think the above would be recommended, since most of your logic isn’t the DAG itself, and the DAG is just the means of contextually triggering it. However, if each node is more of a data check/wait (e.g. we’re on this step un…

Do you avoid things like task sensors? Based off what you described it sounds like an anti pattern if you’re using them. Great description of good orchestration design. Airflow is fairly open ended in how you can construct dags, leading to some interesting results.

Yes, I think you could make an argument for them, but in general it means putting your state sensing into orchestration (local truth) rather than something external (universal truth). As with anything, it does depend on your application though. If you were running something like an ETL, I think it’s generally more appropriate to sense the output of that ETL (data artifact, table, partition, etc) than it is to sense the task itself. It does present some challenges for e.g. cascading backfills, but I think it’s a fine tradeoff in most applications.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#350

Post by willturman »

1oooqooq wrote: post #132
willturman wrote: post #95

I implemented RabbitMQ based messaging queues as a mechanism to coordinate execution among discrete components of a handful of ambitious laboratory automation systems ~4-8 years ago. Given a recent opportunity to rethink messaging based architectures, I chose the simplicity and flexibility of Redis to implement stack and queue based data-structures accessible across distributed nodes. With even a handful of nodes, it…

sounds like you're about to reinvent a queueing system on top of redis. in a very painful way.

Redis already has the data structures for a queue and stack essentially straight out of the box.
Post reply on HN