Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

271–280 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#271

Post by JeremyNT »

throwaway7ahgb wrote: post #253
alfiedotwtf wrote: post #180

Earlier quoted context omitted.

This! You NEEDED to scale horizontally because machines were just doing too much. I remember when our Apache boxes couldn’t even cope doing SSL so we had a hardware box doing it on ingress!

I used to administrate a small fleet of sun sparc hosts with SSL accelerators. They were so much money $$$$. I proposed dumping them all for a smaller set of x86 hosts running linux, it took 2-3 years before the old admins believed in the performance and cost savings. They refused to believe it would even work.

I lived through that era too, it was wild to see how quickly x86 dethroned sparc (even Intel's big misses like Itanium were only minor bumps in the road).

Those days, you had to carefully architect your infrastructure and design your workload to deal with it, and every hardware improvement required you to reevaluate what you were doing. Hence novel architectural choices.

Everything is way easier for normal sized organizations now, and that level of optimization is just no longer required outside of companies doing huge scale.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#272

Post by zug_zug »

busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

To second this theory with some real-world data. A few startups I worked at a NY scala shop that used tons of Akka (event-driven queuing scala thing). Why? Because a manager at his prior job "saved the company" when "everything was slow" by doing this, so mandated it at the new job?

What were doing that required queueing? Not much, we showed people's 401ks on a website, let them adjust their asset mix, and sent out hundreds of emails per day. As you would expect, people almost never log into their 401k website.

A year or so after working there I realized our servers had been misconfigured all along and basically had 0 concurrency for web-requests (and we hadn't noticed because 2 production servers had always served all the traffic we needed). Eventually we just ripped out the Akka because it was unnecessary and added unnecessary complexity.

In the last month this company raised another funding round with a cash-out option, apparently their value has gone up and they are still doing well!

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#273

Post by appplication »

RangerScience wrote: post #108
appplication wrote: post #93

Earlier quoted context omitted.

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…

Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things…

I think there are some questions to ask that can help drive your system design here. Does each node in the DAG represent an event at which some complex automated logic would happen? If so, then I think the above would be recommended, since most of your logic isn’t the DAG itself, and the DAG is just the means of contextually triggering it.

However, if each node is more of a data check/wait (e.g. we’re on this step until you tell me you completed some task in the real world), then it would seem rather than your DAG orchestrating nodes of logic, the DAG itself is the logic. In this case, i think you have a few options, though Airflow itself is probably not something I would recommend for such a system.

In the case of the latter, there are a lot of specifics to consider in how it’s used. Is this a universal task list, where there is exactly one run of this DAG (e.g. tracking tasks at a company level), or would you have many independent runs of this (e.g. many users use it), are runs of it regularly scheduled (e.g. users run it daily, or as needed).

Without knowing a ton about your specifics, a pattern I might consider could be isolating your logic from your state, such that you have your logical DAG code, baked into a library of reusable components (a la the above), and then allowing those to accept configuration/state inputs that allow them to route logic appropriately. As a task is completed, update your database with the state as it relates to the world, not its place in the DAG. This will keep your state isolated from the logic of the DAG itself, which may or may not be desirable, depending on your objectives and design parameters.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#274

Post by perrygeo »

Message queues are great for flowing information to external systems, one-way. But it's always worth asking why you have separate systems in the first place. And if your app has to also consume from a queue, forming any sort of cycle, look out.

Especially if they are services that are conceptually related, you will immediately hit consistency problems - message queues offer few guarantees about data relationships, and those relationships are key to the correctness of every system I've seen. Treating message queues as an ACID data store is a categorical mistake if you need referential integrity - ACID semantics are ultimately what most business processes need at the end of the day and MQ architectures end up accumulating cruft to compensate for this.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#275

Post by KronisLV »

mewpmewp2 wrote: post #113
KronisLV wrote: post #97

Earlier quoted context omitted.

> Order confirmation emails and sms messages are triggered by the order submission, and again usually sent off to a 3rd party bulk email or SMS service. Twilio or Campaign Monitor or Mailchimp will have queues and retry mechanisms, but again the website devs are just firing off an API call to some 3rd party that's dealing with that. In my case, I need to compile templates for the e-mails to be sent, which is somewhat…

What is taking so long for compiling? And what language are you using? E.g. in NodeJS you can fire of requests without having ti have a new thread or having to wait for it.

> What is taking so long for compiling?

Razor templates: https://learn.microsoft.com/en-us/aspnet/core/mvc/views/razo... with this outside of MVC https://github.com/adoconnection/razorenginecore

Though to be honest, if it wasn't template compilation, it might as well be slow DB queries, slow external service calls, or anything else that you wouldn't want to do in the lifecycle of the user's request, if it concerns something that should happen in the background, not even related to a particular stack.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#276

Post by scotty79 »

EVa5I7bHFq9mnYK wrote: post #187
Dalewyn wrote: post #161

Earlier quoted context omitted.

It's kind of mind boggling just how powerful mundane desktop computers have gotten, let alone server hardware. Think about it: That 20 core CPU (eg: i7 14700K) you can buy for just a couple hundred dollars today would have been supercomputer hardware costing tens or hundreds of thousands of dollars just a decade ago.

According to geekbench, an i9 4790 processor released a decade ago is ~5 times slower than i7 14700. 4790's go for $30 at ebay, vs $300 for 14700, so price/performance seems to be in favor of older hardware:)

I think for server type workloads to get performance improvement estimate it would be reasonable to compare single core performance and multiply by the ratio of number of cores.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#277

Post by grugagag »

bigiain wrote: post #54
grugagag wrote: post #43

Earlier quoted context omitted.

One word: scale. The services you mention above do require scale if commercial. OP argues and I somewhat agree that lots of resume driven tech was oversold and overused making things more complicated and expensive than they should have. Once tech gets more mature it’s harder do misuse and it is used where real needs arise.

This is true, but in my opinion badly misunderstood. There are a huge number of "commercial" things that are hitting several million dollars a month in revenue running Ruby On Rails or WordPress/PHP. You can scale a long long way with "boring technology". Way too many people think that are "the unicorn" who's user base is going to grow so quickly that then need billion simultaneous user scale right now - instead of r…

I absolutely agree. But keep in mind that a lot of these services are made by startups that want to be acquired by someone with big pockets. Using hyped tech helps the sale. Don’t ask me why…

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#278

Post by robertclaus »

I find it super interesting that the comments calling out "obviously we all still use message queues and workers, we just don't write about them" are buried half way down the comments section by arguments about Microservices and practical scalability. A junior engineer reading the responses could definitely get the false impression that they shouldn't offload heavy computation from their web servers to workers at all anymore.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#279

Post by busterarm »

nailer wrote: post #240
busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

> - Start new projects in 2024 on the latest RabbitMQ version and try to use classic queues I am out of the loop here. Did Rabbit break classic queues?

Deprecated versions/years ago and set to be removed in the next release.

Also they're being used in a case where reliability is a must, thus they shouldn't be using classic queues.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#280

Post by Rury »

It's the same reason why any old fad isn't in vogue anymore - it's just how popular things go. I mean, message queues weren't exactly new things in the late 2000s... your standard GUI and mouse uses message queues, pretty much since the 1980s. More and more people just caught on over time, popularity hit a peak, and then people eventually moved on. They're still used in many places, just no longer what's being crazed about.
Post reply on HN