Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

291–300 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#291
post #234
post #211

Earlier quoted context omitted.

« Use kafka as a basic message queue ». Since i’m guilty of that (i use kafka as the backbone for pretty much any service 2 service communication, under a « job » api) i wonder why you think that’s wrong.

you have easiest ways to do a basic message queue. Kafka is overkill and it doesn't worth the overhead in simple escenarios.

This. And as-mentioned, RabbitMQ is already part of our platform. Using Kafka here is both extremely wasteful and unnecessarily complicates the product.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#292
post #213

Earlier quoted context omitted.

Machine failures are few and far between these days. Over the last four years I've had a cluster of perhaps 10 machines. Not a single hardware failure. Loads of software issues, of course. I know this is just an anecdote, but I'm pretty certain reliability has increased by one or two orders of magnitude since the 90s.

You didn't answer the question though. You're answer is "it won't" and that isn't a good strategy.

It is in that if something happens less often, you don't need to prepare for it as much if the severity stays the same (cue in Nassim Taleb entering the conversation).

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#293
post #292

Earlier quoted context omitted.

You didn't answer the question though. You're answer is "it won't" and that isn't a good strategy.

It is in that if something happens less often, you don't need to prepare for it as much if the severity stays the same (cue in Nassim Taleb entering the conversation).

I'm not sure what types of products you work on, but it's kind of rare at most companies I've worked at where having a backup like that is a workable solution.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#294
post #230

Earlier quoted context omitted.

What happens when the single machine fails?

Worst case scenario you service is not available for a couple of hours. In 99% of business, customers are totally okay with that (if it's just not every week). IRL shops are also occasionally closed due to incidents; heck even ATMs and banks don't work 100% of the time. And that's the worst case: because your setup is so simple, restoring a backup or even doing a full setup of a new machine is quite easy. Just make s…

Depending on your product, this could mean tens of thousands to millions of dollars worth of revenue loss. I don't really see how we've gone backwards here.

You could just distribute your workloads using...a queue, and not have this problem, or have to pay for and pay to maintain backup equipment etc.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#295
post #148
post #18

My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services. These days all those pe…

I have also seen a lot of cases where engineers would use the more unnecessarily complex structure on purpose to make themselves less replaceable, as it would take longer time for newcomers to get familiar with the environment deployed.

The reality is that management never cares how non-replaceable an engineer is, fires them anyway, a bunch of stuff breaks and the newcomers are stuck holding the bag.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#296
post #24

The web got faster, and it became easier to build and consume APIs, so we eliminated the need for an intermediary. More "native" event-driven architectures emerged.

"The web got faster" [citation needed] Sure the network has gotten faster, but with the few exceptions like Craigslist and this site, in general "the web" has gotten way way slower in real terms, at least as I see it. These days megabytes of javascript (or webassembly) seems to be required just to display pages with a kilobyte of two of actual text content...

I should have clarified that I mean the backend infrastructure is much faster; I'm not talking about the client's experience (since rarely do end clients interact with message queues)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#297
post #26

The web got faster, and it became easier to build and consume APIs, so we eliminated the need for an intermediary. More "native" event-driven architectures emerged.

It's weird that people say this and yet the web feels very slow.

I should have clarified that I mean the backend infrastructure is much faster; I'm not talking about the client's experience (since rarely do end clients interact with message queues)

(in other words, by "web" I mean the network, not the content)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#298
post #81

I think it's simple: async runtimes/modules in JavaScript/Node, Python (asyncio), and Rust. Those basically handle message queues for you transparently inside of a single application. You end up writing "async" and "await" all over the place, but that's all you need to do to get your MVP out. And it will work fine until you really become popular. And then that can actually still work without external queues etc. if y…

To back up the story regarding async a bit, at least on the front end ... A long time ago in the 2000s, on front-end systems we'd have a server farm to handle client connections, since we did all rendering on the server at the time. On the heavyweight front end servers, we used threading with one TCP connection assigned to each thread. Threading was also less efficient (in Linux, at least) than it is now, so a large number of clients necessitated a large number of servers. When interfacing with external systems, standard protocols and/or file formats were preferred. Web services of some kind were starting to become popular, usually just when interfacing with external systems, since they used XML(SOAP) at the time and processing XML is computationally expensive. This was before Google V8 was released, so JavaScript was seen as sort of a (slow) toy language to do only minor DOM modifications, not do significant portions of the rendering. The general guidance was that anything like form validation done on the client side in JS was to be done only for slight efficiency gains and all application logic had to be done on the server. The release of NGINX to resolve the C10K problem, Google V8 to make JS run faster, and Node.js to scale front end systems for large numbers of idle TCP connections (C10k) all impacted this paradigm in the 2000s.

Internally, applications often used proprietary communication protocols, especially when interacting with internal queueing systems. For internal systems, businesses prefer data be retained and intact. At the time, clients still sometimes preferred systems be able to participate in distributed two-phase commit (XA), but I think that preference has faded a bit. When writing a program that services queues, you didn't need to worry about having a large number of threads or TCP connections -- you just pulled a request message from the request queue, processed the message, pushed a response onto the response queue, and moved on to the next request message. I'd argue that easing the strong preference for transactional integrity, the removal of the need for internal services to care about the C10k problem (async), and the need to retain developers that want to work with recent "cool" technologies reduced the driver for internal messaging solutions that guarantee durability and integrity of messages.

Also, AWS's certifications try to reflect how their services are used. The AWS Developer - Associate still covers SQS, so people are still using it, even if it isn't cool. At my last job I saw applications using RabbitMQ, too.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#299
post #143

Earlier quoted context omitted.

Well, the one thing they have in common is that both projects were originally designed by the same person, Pieter Hintjens.

He was involved in the AMQP mess, not in designing RabbitMQ.

You are correct, I misremembered.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#300
post #191
post #131

Earlier quoted context omitted.

If you are using Chrome, it's also a combination of multiple well separated processes talking via RPC with each other, which is pretty similar to microservices, although the separation boundaries are more influenced by attack mitigation requirements than your typical microservice architecture would be.

But that’s due to security, not for any supposed benefit of microservices. Also, both processes are from the same repo sharing code, so I wouldn’t really qualify as microservice.

1. microservices also create security boundaries

2. microservices living in monorepos is common

Post reply on HN