Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

281–290 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#281

Post by busterarm »

janstice wrote: post #226
busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

To be fair, thousands of microservices in a monorepo sounds better than thousands of microservices, each with its own repository (but sub-repo’ing common libraries so everything jams up in everything else).

When your monorepo grows into the dozens upon dozens of gigabytes in size, the effort to manage it grows exponentially and the tooling available to you to interact with it shrinks exponentially. For example, your options for CI/CD systems and code review tools to use are extremely limited.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#282

Post by peoplefromibiza »

ElevenLathe wrote: post #258
_xiaz wrote: post #251

Earlier quoted context omitted.

In defense of CDNs they're also pretty neat for cutting down latency, which benefits even the first customer. Of course that only helps if you don't end up shoving dozens of MBs in Javascript/JSON over the wire.

Putting your app behind a CDN also gives you some cheap defense against (most, casual) DDoS.

usually, but this app is not even exposed publicly to the internet

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#283

Post by dchftcs »

bigiain wrote: post #46
abeppu wrote: post #33

Earlier quoted context omitted.

I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should…

Tracking, visitor analytics, and observability type things are all (in general) going out to 3rd party specialist services for those things, and getting dropped into a time series database (or, for us old school gray beards, a log file) and processed later. It's rare for the website devs to be doing anything more complex that adding in even more javascript to their pages for those, no need to message queues for that.…

There's always a queue somewhere at some level of resilience. Sometimes it's as mundane as the TCP buffer, some other times the other end points may become unresponsive for a while, including the time series DB you seem to want to take as the source of truth, or whatever that tries to resend and reconcile the requests.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#284

Post by bamboozled »

perbu wrote: post #213
bamboozled wrote: post #208

Earlier quoted context omitted.

What happens when the single machine fails?

Machine failures are few and far between these days. Over the last four years I've had a cluster of perhaps 10 machines. Not a single hardware failure. Loads of software issues, of course. I know this is just an anecdote, but I'm pretty certain reliability has increased by one or two orders of magnitude since the 90s.

You didn't answer the question though. You're answer is "it won't" and that isn't a good strategy.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#285

Post by busterarm »

zug_zug wrote: post #272
busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

To second this theory with some real-world data. A few startups I worked at a NY scala shop that used tons of Akka (event-driven queuing scala thing). Why? Because a manager at his prior job "saved the company" when "everything was slow" by doing this, so mandated it at the new job? What were doing that required queueing? Not much, we showed people's 401ks on a website, let them adjust their asset mix, and sent out h…

I think I'm actually somewhat familiar with your story.

There's something about "Java/Scala", "New York startup" and cargo-culted behaviors. I'm sure this happens elsewhere too in other circles, but I've both heard of and read about what you're referring to before.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#286

Post by busterarm »

SwissCoreWizard wrote: post #237
busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

> I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America and for almost every engineer here this is their first job. Are you making this up to make your claim sound more extravagant? Even Citadel is not that picky so unless you work at OpenAI/Anthropic I'm calling nonsense.

There's nothing about the company or the product that requires it but we in fact are that picky when it comes to hiring new grads. Only new grads will work here at the pay we're offering and for all the other reasons mentioned though, so there you go.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#287

Post by busterarm »

spacebanana7 wrote: post #196
busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

Raw intelligence is of limited help when working in an area that requires lots of highly depreciating domain specific knowledge. Relatively few graduates know their way around the Snowflake API, or the art of making an electron app not perform terribly. Even sending an email on the modern internet can require a lot of intuition and hidden knowledge. > There's over 30 microservices in our org to every 1 engineer I won…

> > There's over 30 microservices in our org to every 1 engineer

> I wonder if this a factor in making onboarding of new hires difficult?

Not so much that but a culture of rapidly reconfiguring the whole organization around new product work every three sprints and no ownership/responsibility for maintaining prior work.

It almost seems obvious what would happen.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#288

Post by magicalhippo »

spacebanana7 wrote: post #244
magicalhippo wrote: post #221

Earlier quoted context omitted.

> I wonder if this a factor in making onboarding of new hires difficult? Surely it's because they focused on making onboarding easy? New hires just make a new microservice for every task they get, don't need to know the broader scope just the specs for the service.

It's hard to make code changes without understanding the broader scope. For example, a developer asked to change the payments processor from the Stripe API to the Adyen API needs to know about: 1) credential scoping across microservices and test environments 2) impacts on billing, tax management and customer support 3) the organisation's approach to external dependencies (e.g do they use some kind of internal registr…

My point is with microservices you can, in theory, write all that stuff in a spec, and hand it to some new kid on the block. They then don't need to know about billing or what not, you've taken care of that in the spec.

Not saying it's a good way, but it's one way to end up with an inordinate amount of microservices per dev.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#289

Post by hn_throwaway_99 »

steve1977 wrote: post #109
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> because people realized "You're not Google. Your company will never be Google." Is that also why almost no one is using microservices and Kubernetes?

I realize you're being sarcastic (I think), but I actually would put microservices in the same boat. There was a huge push to microservices in the mid teens, and a lot of companies came to hugely regret it. There is a reason this video, https://youtu.be/y8OnoxKotPQ, is so popular.

And it's not that "no one is using microservices", it's just that tons of companies realized they added almost as many complications as they alleviated, and that for many teams they were just way too premature. And a lot of the companies that I've seen have the most success with microservices are also the most pragmatic about them: they use them in some specific, targeted areas (e.g. authn and authz), but otherwise they're content using a well-componentized monolith where they can break off independent services later if there is an explicit reason to do so.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#290

Post by chadrstewart »

So others have mentioned this in previous posts that 99% of companies building an app will not actually need this level of infrastructure especially considering how much better computers have gotten since the 2000s.

But...

Isn't another reason why we don't see hype around message queues and such in Distributed Systems because they are standard practice now? This discourse feels around this feels like, "message queues will make your architecture so much better, you should try it!" and more like, "just use a message queue..." To feel like the hype isn't there anymore because the technology is just standard practice now.

I could be wrong but whenever I come across any articles on building Distributed Systems, message queues and their variants are one of the first things mentioned as a necessity.

Post reply on HN