Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

361–370 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#361

Post by morbicer »

burutthrow1234 wrote: post #62

I think "message queues" have become pretty commoditized. You can buy Confluent or RedPanda or MSK as a service and never have to administer Kafka yourself. Change Data Capture (CDC) has also gotten really good and mainstream. It's relatively easy to write your data to a RDBMS and then capture the change data and propagate it to other systems. This pattern means people aren't writing about Kafka, for instance, becaus…

This. You don't need Google scale for Kafka to make sense. Just few acquisitions and a need to fan out some data to multiple products. For example you have SCIM hooks that will write to Kafka so all parts of org can consume the updates. Or customer provisioning.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#362

Post by Noneofya »

appplication wrote: post #93
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…

Thanks, this was really helpful.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#363

Post by Noneofya »

RangerScience wrote: post #108
appplication wrote: post #93

Earlier quoted context omitted.

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…

Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things…

Tbh that sounds almost like an already built workflow engine like n8n or even Jira would be preferable to reinventing the wheel.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#364

Post by Noneofya »

kypro wrote: post #41
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

>My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to.

I generally agree with your stance and I would add that I find the whole simplistic „microservices suck“ talking point as nonsensical as viewing them as a panacea. They do solve a few specific (for most companies, mostly organizational/human factors because scale and redundancy don’t matter that much) problems that are harder to solve with monoliths.

I still think this point is a bit misleading, because yes, the components become simpler, but their interaction becomes more complex and that complexity is now less apparent. See:

>The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith.

I think that perspective often doesn’t match reality. Maybe most microservice domains I‘ve seen had poorly separated domains, but changes small enough to ignore the interactions with other services have been rare in my experience.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#365

Post by Noneofya »

kaba0 wrote: post #191
jupp0r wrote: post #131

Earlier quoted context omitted.

If you are using Chrome, it's also a combination of multiple well separated processes talking via RPC with each other, which is pretty similar to microservices, although the separation boundaries are more influenced by attack mitigation requirements than your typical microservice architecture would be.

But that’s due to security, not for any supposed benefit of microservices. Also, both processes are from the same repo sharing code, so I wouldn’t really qualify as microservice.

That‘s literally an example of the decision depending on multiple factors. Separation of concerns -> more isolation -> stronger security of the overall system is exactly one of the possible benefits of microservices.

Scale is just one. There is also fault tolerance, security, organizational separation (which can be, up to a point, also be realized with libraries as you suggest), bigger ecosystem to choose from, …

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#366

Post by o999 »

oceanplexian wrote: post #295
o999 wrote: post #148

Earlier quoted context omitted.

I have also seen a lot of cases where engineers would use the more unnecessarily complex structure on purpose to make themselves less replaceable, as it would take longer time for newcomers to get familiar with the environment deployed.

The reality is that management never cares how non-replaceable an engineer is, fires them anyway, a bunch of stuff breaks and the newcomers are stuck holding the bag.

All fun and games until stock price tanks because of shitty uptime

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#367

Post by XCSme »

hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

So true, people optimize prematuriley for when they'll have 100m monthly users, when they have no product yet and won't likely reach 100k users in many years (which can run on a single $100/m dedicated machine...).

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#368

Post by tamiral »

We highly use messaging queues to power millions of daily transactions and requests. I think it’s because the patterns have been written about and it is no longer a fun hot topic. AWS SQS, Azure Event Hub etc all very standard for what I’ve seen in recent architectural diagrams from many companies.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#369

Post by didgetmaster »

ithkuil wrote: post #156
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

Hardware has certainly progressed significantly over the years, but the size of workloads has also grown.

The big question is: Does a 'reasonable' workloads today fit on a single machine better than a 'reasonable' workload did 20 years ago?

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#370

Post by taylodl »

The technology just got mature enough that it's not exciting to write about, but it's still really widely used.

Message queue-based architectures are the backbone of distributed, event-driven systems. Think of systems where when this particular event happens, then several downstream systems need to be aware and take action. A message queue allows these systems to be loosely coupled and supports multiple end system integration patterns.

Notice, this is systems or enterprise development, not application development. If you're using a message queue as part of your application architecture, then you may be at risk of over-engineering your solution.

Post reply on HN