Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

241–250 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#241
post #209

Earlier quoted context omitted.

What's wrong with having your infrastructure as code and storing it in Git?

YAML isn't code. Same as your reply, YAML has very little awareness of context.

In computer science, the word "code" has a very specific meaning, and markup languages definitely fit this definition.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#242
post #210

Earlier quoted context omitted.

What's wrong with having your infrastructure as code and storing it in Git?

Nothing. Even if it's objectively terrible (thousands of lines of templated YAML, or thousands of lines of spaghetti bash), being in Git as code is still better. At least you know what it is, how it evolved, and can start adding linting/tests.

I manage large IaC repos and it's mostly HCL, well-structured and easy to work with. Where we have Kubernetes manifests, they're usually split into smaller files and don't cause any trouble as normally we usually don't deploy manifests directly.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#243

Earlier quoted context omitted.

Define "no one". If you mean small shops, maybe. If you mean large organizations, I haven't seen even one in the last 5 years that wouldn't use them in one way or another.

It was meant to be ironic

Ah sorry, it is more and more difficult for me to detect irony these days.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#244

Earlier quoted context omitted.

Raw intelligence is of limited help when working in an area that requires lots of highly depreciating domain specific knowledge. Relatively few graduates know their way around the Snowflake API, or the art of making an electron app not perform terribly. Even sending an email on the modern internet can require a lot of intuition and hidden knowledge. > There's over 30 microservices in our org to every 1 engineer I won…

> I wonder if this a factor in making onboarding of new hires difficult? Surely it's because they focused on making onboarding easy? New hires just make a new microservice for every task they get, don't need to know the broader scope just the specs for the service.

It's hard to make code changes without understanding the broader scope.

For example, a developer asked to change the payments processor from the Stripe API to the Adyen API needs to know about:

1) credential scoping across microservices and test environments 2) impacts on billing, tax management and customer support 3) the organisation's approach to external dependencies (e.g do they use some kind of internal registry)

Microservices don't inherently make this kind of change more difficult. A well isolated payments microservice might even help. But having hundreds or thousands of microservices spread across several code repos makes it quite difficult to figure out how your change will affect other services.

I acknowledge I've picked an extreme example that wouldn't be asked of a new hire. However, smaller versions of the same challenges will happen in the smaller API upgrades & migrations that make up a big part of what the average CRUD developer does.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#245
My company heavily relies on Amazon SQS for background jobs. We use Redis as well but it is hard to run at scale. Hence, anything critical goes to SQS by default. SQS usage is so ubiquitous I can’t imagine anyone be interested in writing a blog post or presenting on a conference. Once you get used to SQS specifics (more than once delivery, message size limit, client/server tooling built, expiration settings, DLQ) I doubt there’s anything that can beat it in terms of performance/reliability Unless you have resources to run Redis/Kafka/etc yourself. I would recommend searching for talks by Shopify eng folks on their experience, in particular from Kir (e.g. https://kirshatrov.com/posts/state-of-background-jobs)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#247
At least what I have seen is that MQ was mostly used for batches in the areas I worked in.

The new thing is "event driven architecture" (or whatever they can pass off as that hype). In a lot of cases, it's a better architecture. For fhe remaining batches, we are running against S3 buckets, or looking at no SQL entries in a specific status in a DB. And we still use a little SQS, but not that often.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#248

Earlier quoted context omitted.

It was meant to be ironic

Ah sorry, it is more and more difficult for me to detect irony these days.

Yeah sorry as well, I actually wanted to add a “… oh wait” to my original comment but forgot to do it… (too busy fixing a podman issue… )

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#249
Regarding this issue, I have some observations of my own. I've noticed that systems based on queues, such as Kafka, AMQP, etc., are still very widespread, for example in vehicle networking, transaction systems, and so on. I recently encountered a customer deploying Kafka on AWS, with monthly consumption of Kafka-related computing storage exceeding $1 million. The cluster scale is huge, containing various system events, logs, etc. I've also seen customers building IoT platforms based on Kafka. Kafka has become very central to the IoT platform, and any problems can cause the entire IoT platform to be unavailable. I personally have written over 80% of the code for Apache RocketMQ, and today I have created a new project, AutoMQ (https://github.com/AutoMQ/automq). At the same time, we also see that competition in this field is very fierce. Redpanda, Confluent, WarpStream, StreamNative, etc., are all projects built based on the Kafka ecosystem. Therefore, the architecture based on message queues has not become obsolete. A large part of the business has transformed into a Streaming form. I think Streaming and MQ are highly related. Streaming leans more towards data flow, while MQ leans more towards individual messages.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#250
post #41

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

> My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to.

You can achieve exactly the same with simple individual libraries.

Post reply on HN