Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#191

Post by kaba0 »

jupp0r wrote: post #131
wg0 wrote: post #68

Earlier quoted context omitted.

I think lots of microservices can be replaced with a monolith which in turn can be replaced with a set of composable libraries versioned separately. If anyone doubts that, this very browser used to read and write is built all the way up with dozens of libraries from compression, network, image encoding, decoding, video encoding decoding, encryption, graphics, sound and what not where each library is totally separate…

If you are using Chrome, it's also a combination of multiple well separated processes talking via RPC with each other, which is pretty similar to microservices, although the separation boundaries are more influenced by attack mitigation requirements than your typical microservice architecture would be.

But that’s due to security, not for any supposed benefit of microservices. Also, both processes are from the same repo sharing code, so I wouldn’t really qualify as microservice.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#192

Post by mcqueenjordan »

I think it's a mix of:

1. Queues are actually used a lot, esp. at high scale, and you just don't hear about it.

2. Hardware/compute advances are outpacing user growth (e.g. 1 billion users 10 years ago was a unicorn; 1 billion users today is still a unicorn), but serving (for the sake of argument) 100 million users on a single large box is much more plausible today than 10 years ago. (These numbers are made up; keep the proportions and adjust as you see fit.)

3. Given (2), if you can get away with stuffing your queue into e.g. Redis or a RDBMS, you probably should. It simplifies deployment, architecture, centralizes queries across systems, etc. However, depending on your requirements for scale, reliability, failure (in)dependence, it may not be advisable. I think this is also correlated with a broader understanding that (1) if you can get away with out-of-order task processing, you should, (2) architectural simplicity was underrated in the 2010s industry-wide, (3) YAGNI.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#193

Post by liampulles »

"The technology just got mature enough that it's not exciting to write about, but it's still really widely used."

My money is on this. I think the simple usecase of async communication, with simple pub/sub messaging, is hugely useful and not too hard to use.

We (as a Dev community) have just gotten over event sourcing, complex networks and building for unnecessary scale. I.e. we're past the hype cycle.

My team uses NATS for Async pub/sub and synchronous request/response. It's a command driven model and we have a huge log table with all the messages we have sent. Schemas and usage of these messages are internal to our team, and are discarded from NATS after consumption. We do at-least-once delivery and message handlers are expected to be idempotent.

We have had one or two issues with misconfiguration in NATS resulting in message replay or missed messages, but largely it has been very successful. And we were a 3 person dev team.

It's the same thing as Kubernetes in my mind - it works well if you keep to the bare essentials and don't try to be clever.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#194

Post by peoplefromibiza »

hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> "You're not Google. Your company will never be Google."

I'm not sure people realize this now more than then. I was there back then and we surely knew we would never be Google hence we didn't need to "scale" the same way they did.

Nowadays every project I start begins with a meeting where is presented a document describing the architecture we are going to implement, using AWS of course, because "auto-scale" right?, and 9/10 it includes CloudFront, which is a CDN and I don't really understand why this app I am developing, which is basically an API gateway with some customization that made Ngnix slightly less than ideal (but still perfect for the job), and that averages to 5 rps needs a CDN in front of it... (or AWS or auto-scaling or AWS lambda, for that matter)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#195

Post by tichiian »

The other answers around here are mostly right, but I'd like to add another one, which is right in some situations:

Message queues are often the wrong tool. Often you rather want something like RPC, and message queues were wrongly used as poor man's async DIY RPC.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#196

Post by spacebanana7 »

busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

Raw intelligence is of limited help when working in an area that requires lots of highly depreciating domain specific knowledge.

Relatively few graduates know their way around the Snowflake API, or the art of making an electron app not perform terribly. Even sending an email on the modern internet can require a lot of intuition and hidden knowledge.

> There's over 30 microservices in our org to every 1 engineer

I wonder if this a factor in making onboarding of new hires difficult?

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#197

Post by Dalewyn »

EVa5I7bHFq9mnYK wrote: post #187
Dalewyn wrote: post #161

Earlier quoted context omitted.

It's kind of mind boggling just how powerful mundane desktop computers have gotten, let alone server hardware. Think about it: That 20 core CPU (eg: i7 14700K) you can buy for just a couple hundred dollars today would have been supercomputer hardware costing tens or hundreds of thousands of dollars just a decade ago.

According to geekbench, an i9 4790 processor released a decade ago is ~5 times slower than i7 14700. 4790's go for $30 at ebay, vs $300 for 14700, so price/performance seems to be in favor of older hardware:)

On the other hand, the E7-8890 v3 (the closest equivalent to a 14700K in core count at the time from a quick glance) had an MSRP of $7174.00[1].

So maybe I was a bit too high on the pricing earlier, but my point still stands that the computing horsepower we have such easy access to today was literal big time magic just a decade ago.

[1]: https://ark.intel.com/content/www/us/en/ark/products/84685/i...

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#198

Post by EVa5I7bHFq9mnYK »

busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

Because JS is single threaded, and an average programmer today is too dumb to learn any language beyond JS, you must build everything into "microservices".

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#200

Post by samus »

benterix wrote: post #178
keredson wrote: post #122

Earlier quoted context omitted.

and mandates that virtually all new projects not directly use borg/kubernetes.

That's interesting. What's the rationale behind that?

I guess such a service gets coupled too strongly to that platform, and major engineering effort is required to deploy it the old-school way.
Post reply on HN