Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#41

Post by kypro »

tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessarily understand all of the domain complexity of the code they're touching.

Monoliths imo are better for smaller projects with a few devs, but otherwise within a few years most of the time you'll regret building a monolith.

I also disagree with the duplicated code point. I don't understand why that would be a significant problem assuming you're using the same language and sharing packages between projects. This isn't a problem I've ever had while working on microservices anyway. I'd also debate whether they're anymore more complex than monoliths on average. My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to. The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#42

Post by democracy »

speed_spread wrote: post #19

Hypothesis: The performance of systems has increased so much that many things that required a queue before can now just be made into regular synchronous transactions. aka PostgreSQL is eating the world.

Nah, messaging based/async operations are rarely for performance benefits...

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#43

Post by grugagag »

abeppu wrote: post #33
bigiain wrote: post #18

My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services. These days all those pe…

I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should…

One word: scale. The services you mention above do require scale if commercial. OP argues and I somewhat agree that lots of resume driven tech was oversold and overused making things more complicated and expensive than they should have. Once tech gets more mature it’s harder do misuse and it is used where real needs arise.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#44

Post by mountainriver »

It’s because they were billed as a way to reduce complexity but in reality just added a ton.

The fundamental issue with event driven architecture is getting out of sync with the source of truth.

Every single design doc I’ve seen in an organization pitching event driven architecture has overlooked the sync issue and ultimately been bitten by it.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#45

Post by airocker »

I think dbs like etcd mongo Postgres all have some notion of events and notifications. The only drawback is that you have to still poll from the ui if you use these. This is sufficient for most use cases. If you want real time updates, Kafka provides long running queries which keep Kafka still relevant.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#46

Post by bigiain »

abeppu wrote: post #33
bigiain wrote: post #18

My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services. These days all those pe…

I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should…

Tracking, visitor analytics, and observability type things are all (in general) going out to 3rd party specialist services for those things, and getting dropped into a time series database (or, for us old school gray beards, a log file) and processed later. It's rare for the website devs to be doing anything more complex that adding in even more javascript to their pages for those, no need to message queues for that.

Order confirmation emails and sms messages are triggered by the order submission, and again usually sent off to a 3rd party bulk email or SMS service. Twilio or Campaign Monitor or Mailchimp will have queues and retry mechanisms, but again the website devs are just firing off an API call to some 3rd party that's dealing with that.

So there are no doubt message queues being used in non-sexy 3rd party services, but those companies probably consider that kind of thing to be their "secret sauce" and don't blog about it.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#48

Post by mindcrime »

Simple: messaging systems / event-driven systems aren't the "fad of the day" anymore, so you don't have a gazillion vendors pumping out blog posts and Youtube videos to hawk their wares, or a million companies writing about them to look "hip" and "cool" to help recruit developers, or endless legions of consulting companies writing about them in order to attract customers, etc.

Basically every "cool, shiny, new" tech goes through this process. Arguably it's all related to the Garnter Hype Cycle[1], although I do think the process I'm describing exists somewhat independently of the GHC.

[1]: https://en.wikipedia.org/wiki/Gartner_hype_cycle

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#49

Post by [deleted] »

guybedo wrote: post #32

Queueing systems haven't disappeared as thery're an important part of distributed systems. If you need to distribute work/data asynchronously to multiple workers, you're gonna use a queuing system. Although queuing systems can be implemented on top of a database, message queues like RabbitMQ / ZeroMQ are doing a fine job. I use RabbitMQ all the time, precisely because i need to transfer data between systems and i hav…

[deleted]

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#50

Post by kaladin_1 »

Personal experience: I needed a message broker while working with multiple sensors constantly streaming data at high frequency.

I have seen a startup where RabbitMQ was being used to hand-off requests to APIs (services) that take long to respond. I argued for unifying queueing and data persistence technology using Postgres even though I know a simple webhook would suffice.

Given that AWS has to sell and complexity tends to make people look smart, another server was spurn up for RabbitMQ :)

Many companies that have run a highly distributed system have figured what works for them. If requests are within the read and write rates of what Redis or Postgres can handle why introduce RabbitMQ or Kafka :?

Always remember that the Engineer nudging you towards more complexity will not be there when the chips are down.

Post reply on HN