Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#101

Post by The_Colonel »

erulabs wrote: post #58
tstrimple wrote: post #52

Earlier quoted context omitted.

Microservices necessarily add more complexity and overhead when compared to a monolith. Just the fact that you have to orchestrate N services instead of just pressing run on a single project demonstrates some of the additional complexity.

Counterpoint: a monolith usually contains a complex init system which allows multiple ways of running the codebase. Microservices can avoid at least that one complexity.

You mean like profiles? Monolith can run like a front service or background worker depending on the config?

In a technical sense it is a complexity, but IME pales in comparison with the alternative of having to manage multiple services.

I actually really like this model, it's pretty flexible. You can still have specialized server instances dedicated for certain tasks, but you have one codebase. What's very sweet is that for local development, you just run a single application which (with all enabled profiles) can fulfill all the roles at the same time.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#102

Post by hn_throwaway_99 »

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google."

That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for 99% of companies. When you couple that with hardware and standard databases getting much better, there are just fewer and fewer companies who need all of these "scalability tricks".

My bar for "Is there a reason we can't just do this all in Postgres?" is much, much higher than it was a decade ago.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#103

Post by dasil003 »

kypro wrote: post #41
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

> My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to.

Whether this is good depends on the type of changes you need to make. Just as you mentioned maintaining modularity in a monolith can be difficult with entropy tending to push the code to spaghetti, there is an equivalent risk in microservices where developers duplicate code or hack around things locally versus making the effort to change the interfaces between microservices where it would make sense.

Ultimately microservices add structure that may be useful for large enough teams, but is still overhead that has to earn its keep.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#104

Post by retrocryptid »

All the devs have already put "event driven" on their resumes. They need something new to be seen to be at the forefront of technology. I think we're in the AI hype phase where everyone is competing for those tasty $500k per year AI jobs at google, so the ACS systems don't know what to do with resumes that have "event driven."

The last thing you want on your LinkedIn profile is a link to a video you made in 2015 about your cool Kafka solution. The ACS would spit you out so fast...

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#105

Post by nosefrog »

hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

To be fair, I worked on multiple projects removing queues at Google, so it's more than just that.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#106

Post by topspin »

nextworddev wrote: post #71

Because it’s massive over engineering of dubious value, unless you are Uber etc

I find it useful in small doses even at small scale. When you have only one of a thing processing stuff it's useful to isolate that behind queues so you can bounce it and lose nothing: "pause" the incoming queue (new messages stack up,) let the pipeline drain, do whatever you must, and then un-pause. Nothing is lost, despite the backend being simple minded and not fault tolerant.

The key here is "small dose". Very limited and strategic use at key of points in the system. Not a wholesale, endemic "messaging architecture."

Queues also provide valuable observability independent of applications: metrics that can be monitored for performance and availability. This is necessary even at small scale. Put "alerts" on the queues: when they start backing up something has likely fallen over.

Queues also help paper over unanticipated conditions, such as Important Customer (tm) fixing/reworking something and suddenly flushing a huge number of messages into my systems: the queue just stacks that up and the rest of the system avoids being overloaded. At the other end, Important Customer neglects their asynchronous consumers, which fail to process 50% of the time. The queue just automatically retries till it goes through.

All of this is transparent to my applications. That has enormous value.

So there you are; real world "grug brained" work that benefits from judicious application of message queues and deftly avoiding the "architecture" tar pit.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#107

Post by darksaints »

It's still around, and it's still going strong in the embedded space. Examples:

* PX4

* Ardupilot

* Betaflight

* DroneCAN

* Cyphal

* ROS/ROS2

* Klipper

* GNU Radio

Also would like to mention that all of the most popular RTOSes implement queues as a very common way of communicating between processes.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#108

Post by RangerScience »

appplication wrote: post #93
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…

Helpful comment! If I could pick your brain...

I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG.

I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things like skipping a node based on some criteria (which, it occurs to me in typing this up, can benefit from your above advice, as that can just be a configured function call that returns skip/no-skip) - and, well... thoughts? (:

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#109

Post by steve1977 »

hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> because people realized "You're not Google. Your company will never be Google."

Is that also why almost no one is using microservices and Kubernetes?

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#110

Post by aspyct »

We have a new project (~ 6 years now) where we implemented a queue with RabbitMQ to temporarily store business events before they are stored in a database for reporting later.

It's awesome!

It absorbs the peaks, smoothes them out, acts as a buffer for when the database is down for upgrades, and I think over all these years we only had one small issue with it.

10/10 would recommend.

Post reply on HN