Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#221

Post by magicalhippo »

spacebanana7 wrote: post #196
busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

Raw intelligence is of limited help when working in an area that requires lots of highly depreciating domain specific knowledge. Relatively few graduates know their way around the Snowflake API, or the art of making an electron app not perform terribly. Even sending an email on the modern internet can require a lot of intuition and hidden knowledge. > There's over 30 microservices in our org to every 1 engineer I won…

> I wonder if this a factor in making onboarding of new hires difficult?

Surely it's because they focused on making onboarding easy?

New hires just make a new microservice for every task they get, don't need to know the broader scope just the specs for the service.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#222

Post by brazzy »

Kirth wrote: post #166
steve1977 wrote: post #115

Earlier quoted context omitted.

That‘s nice, but the question is if you (i.e. your company) needs this ecosystem.

Genuine question: say you have 3-4 services and a bunch of databases that make up your product, what's the alternative to plemping them all into K8s according to you?

Docker compose is another option.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#223

Post by csunbird »

appplication wrote: post #93
tuckerconnelly wrote: post #3

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…

This is exactly what we do, but with Spark instead. We develop the functions locally in a package and call necessary functions for the job notebooks, and the job notebooks are very minimalistic because of this

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#224

Post by perlgeek »

I maintain and develop a message queue-based architecture at work (started around 2014), so here's my take:

* message queues solve some problems that are nowadays easily solved by cloud or k8s or other "smart" infrastructure, like service discovery, load balancer, authentication

* the tooling for HTTPS has gotten much better, so using something else seems less appealing

* it's much easier to get others to write a HTTPS service than one that listens on a RabbitMQ queue, for example

* testing components is easier without the message queue

* I agree with your point about databases

* the need for actual asynchronous and 1:n communication is much lower than we thought.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#225

Post by fikama »

steve1977 wrote: post #109
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> because people realized "You're not Google. Your company will never be Google." Is that also why almost no one is using microservices and Kubernetes?

Maybe add /s ;). It may decline number of hot headed responses

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#226

Post by janstice »

busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

To be fair, thousands of microservices in a monorepo sounds better than thousands of microservices, each with its own repository (but sub-repo’ing common libraries so everything jams up in everything else).

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#227

Post by d0mine »

Copenjin wrote: post #123

I really hope that people are slowly starting to understand that using kafka and turning it in a single point of failure (yes, it fails) of your architecture is not a good idea. This pattern has it's uses, but if you are using it everywhere, every time you have some sort of notifications because "it's easy" or whatever, you are likely doing it wrong and you will understand this at some point and it will not be pleasa…

It is impressive if kafka is the weakest link in your system (99.99% is achievable and even 99.999% is not impossible with the architecture)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#228

Post by badcppdev »

pcl wrote: post #206
Kirth wrote: post #166

Earlier quoted context omitted.

Genuine question: say you have 3-4 services and a bunch of databases that make up your product, what's the alternative to plemping them all into K8s according to you?

3-4 services and a bunch of databases? Assuming there aren’t any particular scaling or performance requirements, if I were managing something like that, I would almost certainly not use k8s. Maybe systemd on a big box for the services?

I agree with you and I'm always confused when people talk about process isolation as a primary requirement for a collection of small internal services with negligible load.

In addition the overhead and reporting drawbacks of running multiple isolated databases is vastly higher than any advantaged gained from small isolated deployments.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#229

Post by Terr_ »

silverquiet wrote: post #37
abeppu wrote: post #33

Earlier quoted context omitted.

I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should…

Bingo - I work on the backend of a medical system and basically anything that interacts with a 3rd party gets put into a queue so our application doesn't choke immediately when one of them has issues. We also have some uses for it within our system. As far as the question, I was thinking that queues have probably just become a standard aspect of modern distributed systems; it's considered a pretty foundational cloud…

At work we usually integrated with a queue, and then some partners/customers wanted a synchronous flow because the user has to see and pick from data we don't own, and now life is pain.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#230

Post by marhee »

bamboozled wrote: post #208
ithkuil wrote: post #156

Earlier quoted context omitted.

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

What happens when the single machine fails?

Worst case scenario you service is not available for a couple of hours. In 99% of business, customers are totally okay with that (if it's just not every week). IRL shops are also occasionally closed due to incidents; heck even ATMs and banks don't work 100% of the time. And that's the worst case: because your setup is so simple, restoring a backup or even doing a full setup of a new machine is quite easy. Just make sure you test you backup restore system regularly. Simple system also tend to fail much less: I've run a service (with customers paying top euro) that was offline for ~two hours due to an error maybe once or twice in 5 years. Both occurrences were due to a non-technical cause (a bill that wasn't payed - yes this happened, the other one I don't recall). We were offline for a couple of minutes daily for updates or the occasional server crash (a go monolith, crash mostly due to an unrecovered panic), however the reverse proxy was configured to show a nice static image with the text along the lines "The system is being upgraded, great new features are on the way - this will take a couple of minute". I installed this the first week when we started the company with the idea that we would do a live-upgrade system when customers started complaining. Nobody ever complained - in fact customers loved to see we did an upgrade once in a while (although most customers never mentioned having seen the image).
Post reply on HN