Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#81

Post by ilaksh »

I think it's simple: async runtimes/modules in JavaScript/Node, Python (asyncio), and Rust. Those basically handle message queues for you transparently inside of a single application. You end up writing "async" and "await" all over the place, but that's all you need to do to get your MVP out. And it will work fine until you really become popular. And then that can actually still work without external queues etc. if you can scale horizontally such as giving each tenant their own container and subdomain or something.

There are places where you need a queue just for basic synchronization, but you can use modules that are more convenient than external queues. And you can start testing your program without even doing that.

Actually async is being used a lot with Rust also, which can stretch that out to scale even farther with an individual server.

Without an async runtime or similar, you have to invent an internal async runtime, or use something like queues, because otherwise you are blocked waiting for IO.

You may still eventually end up with queues down the line if you have some large number of users, but that complexity is completely unnecessary for getting a system deployed towards the beginning.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#82

Post by snapetom »

primitivesuave wrote: post #61
kypro wrote: post #41

Earlier quoted context omitted.

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

Amazing this was downvoted. The comment starts with "in my experience" and is hardly a controversial perspective. I beg the HN community, stop disincentivizing people from respectively providing a converse opinion, lest this become yet another echo chamber of groupthink.

We've swung back and it's trendy to hate on microservices now, so join in! /s

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#83

Post by [deleted] »

wg0 wrote: post #68
kypro wrote: post #41

Earlier quoted context omitted.

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

I think lots of microservices can be replaced with a monolith which in turn can be replaced with a set of composable libraries versioned separately. If anyone doubts that, this very browser used to read and write is built all the way up with dozens of libraries from compression, network, image encoding, decoding, video encoding decoding, encryption, graphics, sound and what not where each library is totally separate…

[deleted]

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#85

Post by zztop44 »

bigiain wrote: post #24
bdcravens wrote: post #4

The web got faster, and it became easier to build and consume APIs, so we eliminated the need for an intermediary. More "native" event-driven architectures emerged.

"The web got faster" [citation needed] Sure the network has gotten faster, but with the few exceptions like Craigslist and this site, in general "the web" has gotten way way slower in real terms, at least as I see it. These days megabytes of javascript (or webassembly) seems to be required just to display pages with a kilobyte of two of actual text content...

Browsing one of those bloated monstrosities on a nice new MacBook with a 2gbps connection is still a lot faster and nicer than Craigslist on old Celeron at 56k.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#88

Post by matrix87 »

kypro wrote: post #41
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…

Microservices also introduce the issue of maintaining schema compatibility for messages between services which usually leads to additional code in order to maintain backward compatibility

From a technical POV they are good for horizontally scaling different workloads which have a different resource footprint

From my experience, when a company decides to go the microservice route, it's more for the sake of solving an organizational problem (e.g. making team responsibilities and oncall escalation more clear cut) than it is to solve a technical problem. Sometimes they will retroactively cite the technical benefit as the reason for using them, but it feels like more of an afterthought

But in all honesty: microservices are very good at solving this organizational problem. If microservice x breaks, ping line manager y who manages it. Very straightforward

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#89

Post by phkx »

ehnto wrote: post #67

They aren't a general solution and don't really add much to your average application. But there are still instances where they make a lot of sense. What I would need to see required before bothering with a message queue architecture: * High concurrency, atomic transactions * Multiple stages of processing of a message required * Traceability of process actions required * Event triggers that will actually be used requi…

Could you elaborate a little on the traceability part, please? If the different steps of a processing chain are distributed via queues, you rather have an overhead in collecting the information in a central place, I would think.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#90

Post by slt2021 »

mountainriver wrote: post #44

It’s because they were billed as a way to reduce complexity but in reality just added a ton. The fundamental issue with event driven architecture is getting out of sync with the source of truth. Every single design doc I’ve seen in an organization pitching event driven architecture has overlooked the sync issue and ultimately been bitten by it.

can you explain how event driven is getting out of sync?

my understanding is that you pipe data change events right into queue, after each and every change, so in theory your application should reflect all changes in near real-time.

it will only go out of sync if your kafka consumers lag behind producers really really bad, and cannot process what is being produced fast enough

Post reply on HN