Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#61

Post by primitivesuave »

kypro wrote: post #41
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

Amazing this was downvoted. The comment starts with "in my experience" and is hardly a controversial perspective. I beg the HN community, stop disincentivizing people from respectively providing a converse opinion, lest this become yet another echo chamber of groupthink.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#62

Post by burutthrow1234 »

I think "message queues" have become pretty commoditized. You can buy Confluent or RedPanda or MSK as a service and never have to administer Kafka yourself.

Change Data Capture (CDC) has also gotten really good and mainstream. It's relatively easy to write your data to a RDBMS and then capture the change data and propagate it to other systems. This pattern means people aren't writing about Kafka, for instance, because the message queue is just the backbone that the CDC system uses to relay messages.

These architectures definitely still exist and they mostly satisfy organizational constraints - if you have a write-once, read-many queue like Kafka you're exposing an API to other parts of the organization. A lot of companies use this pattern to shuffle data between different teams.

A small team owning a lot of microservices feels like resume-driven developnent. But in companies with 100+ engineers it makes sense.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#63

Post by ipsum2 »

democracy wrote: post #39

I think this: "* The technology just got mature enough that it's not exciting to write about, but it's still really widely used." Messaging-based architecture is very popular

Yeah, this is the most likely reason.

It used to be popular to post about rewriting angular to react. Now everyone just uses react (or they write posts about rewriting react to vue or whatever the flavor of the month)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#64

Post by devoutsalsa »

erulabs wrote: post #58
tstrimple wrote: post #52

Earlier quoted context omitted.

Microservices necessarily add more complexity and overhead when compared to a monolith. Just the fact that you have to orchestrate N services instead of just pressing run on a single project demonstrates some of the additional complexity.

Counterpoint: a monolith usually contains a complex init system which allows multiple ways of running the codebase. Microservices can avoid at least that one complexity.

Another advantage of microservices is that you can avoid the overhead of multiple services by having one really big microservice.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#65

Post by weitendorf »

memset wrote: post #12

It may be that lambdas (cloud functions, etc) have become more popular and supported on other platforms. When you enqueue something, you eventually need to dequeue and process it. A lambda just does that in a single call. It also removes the need to run or scale a worker. I think Kafka continues to be popular because it is used as a temporary data store, and there is a large ecosystem around ingesting from streams. I…

This is a big part of it IMO. When your downstream consumers can scale up and down quickly, you don’t necessarily need anything in the middle to smooth out load unless your workloads are especially spiky.

I think this also speaks to a related phenomenon where there are simply more tools and technologies you can buy or run “off the shelf” now. Back in the 2010s everybody was trying to roll their own super complex distributed systems. Nowadays you have a ton of options to pay for more or less polished products to handle that mess for you. No need for engineering meetups and technical blogs about tools that kinda-sorta work if you really know what you’re doing - just pay snowflake or confluent and work on other problems.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#66

Post by vishnugupta »

Speaking from my own experience message queues haven’t disappeared as much as have been abstracted away. For example enqueue to SQS + poll became invoke server less process. There is a message queue in there somewhere just that it’s not as exposed.

Or take AWS SNS which IMO is one level of abstraction higher than SQS. It became so feature rich that it can practically replace SQS.

What might have disappeared is those use cases which used Queues to handle short bursts of peak traffic?

Also streaming has become very reliable tech so a class of usecases that used Queues as streaming pipe have migrated to the streaming proper.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#67

Post by ehnto »

They aren't a general solution and don't really add much to your average application. But there are still instances where they make a lot of sense.

What I would need to see required before bothering with a message queue architecture:

* High concurrency, atomic transactions

* Multiple stages of processing of a message required

* Traceability of process actions required

* Event triggers that will actually be used required

* Horizontal scaling actually the right choice

* Message queues can be the core architecture and not an add on to a Frankenstein API

Probably others, and yes you can achieve all of the above without message queues as the core architecture but the above is when I would think "I wonder if this system should be based on async message queues".

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#68

Post by wg0 »

kypro wrote: post #41
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

I think lots of microservices can be replaced with a monolith which in turn can be replaced with a set of composable libraries versioned separately.

If anyone doubts that, this very browser used to read and write is built all the way up with dozens of libraries from compression, network, image encoding, decoding, video encoding decoding, encryption, graphics, sound and what not where each library is totally separate and sometimes was never intended to be used to build web browsers by the original authors.

Rest assured, most of the business (or web 2.0 systems, search, curate, recommend, surface etc kind of) systems are a lot more simpler then an A class browser.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#69

Post by hnaccountme »

My sense is the hype just died down. There are genuine cases where message queues are the correct solution. But since most of the developers are clueless they just jump on the latest trend and write blog posts and make Youtube videos.

This effect has also happened with microservices/monaliths, lambda/serverless, agile/scrum(still no concrete definition on these). Even cloud as a whole, there are so many articles of how companies managed to cut cloud cost to a fraction just by going bare metal.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#70

Post by kyleyeats »

primitivesuave wrote: post #61
kypro wrote: post #41

Earlier quoted context omitted.

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

Amazing this was downvoted. The comment starts with "in my experience" and is hardly a controversial perspective. I beg the HN community, stop disincentivizing people from respectively providing a converse opinion, lest this become yet another echo chamber of groupthink.

It's because it didn't loop back to queues at any point. It's just a tangent on a tired topic.
Post reply on HN