Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#211

Post by bsaul »

busterarm wrote: post #162

Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub). I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America a…

« Use kafka as a basic message queue ». Since i’m guilty of that (i use kafka as the backbone for pretty much any service 2 service communication, under a « job » api) i wonder why you think that’s wrong.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#212

Post by ndthrowaway »

sofixa wrote: post #140
k8sToGo wrote: post #124

Earlier quoted context omitted.

Need no. But it’s nice especially since I have the know how. Creating a new deployment is just super fast. Developers can deploy their own apps etc. And then there is Helm. If we only decide by “need” then most of the time we also wouldn’t need object oriented programming.

> And then there is Helm Right, who doesn't want to template hundreds of lines of code in a language that uses whitespace for logic and was never made neither for templating nor complex long documents(YAML)? What could possibly go wrong ("error missing xxx at line 728, but it might be a problem elsewhere").

"in a language that uses whitespace for logic"

This argument kind of died when python became one of the most popular programming languages used.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#213

Post by perbu »

bamboozled wrote: post #208
ithkuil wrote: post #156

Earlier quoted context omitted.

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

What happens when the single machine fails?

Machine failures are few and far between these days. Over the last four years I've had a cluster of perhaps 10 machines. Not a single hardware failure.

Loads of software issues, of course.

I know this is just an anecdote, but I'm pretty certain reliability has increased by one or two orders of magnitude since the 90s.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#214

Post by adammarples »

salomonk_mur wrote: post #171
flir wrote: post #150

Earlier quoted context omitted.

Ok, so question (because I really like the DAG approach in principle but don't have enough experience to have had my fingers burned yet): The way you use Airflow, what advantage does it have over crontab? Or to put it another way, once you remove the pipeline logic, what's left?

Observability is a huge upside.

Backfilling is also very useful

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#215

Post by venv »

bamboozled wrote: post #208
ithkuil wrote: post #156

Earlier quoted context omitted.

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

What happens when the single machine fails?

Depending on your requirements for uptime, you could have a stand-by machine ready or you spin up a new one from backups.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#216

Post by inglor_cz »

EVa5I7bHFq9mnYK wrote: post #187
Dalewyn wrote: post #161

Earlier quoted context omitted.

It's kind of mind boggling just how powerful mundane desktop computers have gotten, let alone server hardware. Think about it: That 20 core CPU (eg: i7 14700K) you can buy for just a couple hundred dollars today would have been supercomputer hardware costing tens or hundreds of thousands of dollars just a decade ago.

According to geekbench, an i9 4790 processor released a decade ago is ~5 times slower than i7 14700. 4790's go for $30 at ebay, vs $300 for 14700, so price/performance seems to be in favor of older hardware:)

What about power consumption? When running a server 24/7, power is likely to be a bigger cost concern than the one-off cost of purchasing the processor.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#217

Post by k8sToGo »

sofixa wrote: post #140
k8sToGo wrote: post #124

Earlier quoted context omitted.

Need no. But it’s nice especially since I have the know how. Creating a new deployment is just super fast. Developers can deploy their own apps etc. And then there is Helm. If we only decide by “need” then most of the time we also wouldn’t need object oriented programming.

> And then there is Helm Right, who doesn't want to template hundreds of lines of code in a language that uses whitespace for logic and was never made neither for templating nor complex long documents(YAML)? What could possibly go wrong ("error missing xxx at line 728, but it might be a problem elsewhere").

I don’t like helm itself. But I was referring to the deployment part. I like Kustomize more.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#218

Post by ehnto »

phkx wrote: post #89
ehnto wrote: post #67

They aren't a general solution and don't really add much to your average application. But there are still instances where they make a lot of sense. What I would need to see required before bothering with a message queue architecture: * High concurrency, atomic transactions * Multiple stages of processing of a message required * Traceability of process actions required * Event triggers that will actually be used requi…

Could you elaborate a little on the traceability part, please? If the different steps of a processing chain are distributed via queues, you rather have an overhead in collecting the information in a central place, I would think.

That is certainly true, but message queues are pretty well suited to tagging and audit logs. So the systems involved tag the messages and an audit repository can accept logs from all services.

I guess this isn't in contrast to a monolith which is easier to log, but in contrast to non-message based microservices I think it's easier to implement the audit logging.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#219

Post by k8sToGo »

Kirth wrote: post #166
steve1977 wrote: post #115

Earlier quoted context omitted.

That‘s nice, but the question is if you (i.e. your company) needs this ecosystem.

Genuine question: say you have 3-4 services and a bunch of databases that make up your product, what's the alternative to plemping them all into K8s according to you?

In my home environment I run a VM with docker for that.

In a commercial environment I’d still use kubernetes. But maybe something like k3s or if we are in a cloud environment something like EKS.

Usually with time other services get added to the stack (elastic, grafana, argocd, …)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#220

Post by oblio »

ndthrowaway wrote: post #212
sofixa wrote: post #140

Earlier quoted context omitted.

> And then there is Helm Right, who doesn't want to template hundreds of lines of code in a language that uses whitespace for logic and was never made neither for templating nor complex long documents(YAML)? What could possibly go wrong ("error missing xxx at line 728, but it might be a problem elsewhere").

"in a language that uses whitespace for logic" This argument kind of died when python became one of the most popular programming languages used.

Yeah, but Python has actual for loops.
Post reply on HN