Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#331

Post by koliber »

bamboozled wrote: post #208
ithkuil wrote: post #156

Earlier quoted context omitted.

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

What happens when the single machine fails?

Your monitoring system alerts you on your phone, and you fix the issue.

When I worked with small firms who used kubernetes, we had more kubernetes code issues that machines failing. The solution to the theoretical problem was the cause of real issues. It was expensive to keep fixing this.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#332

Post by koliber »

bamboozled wrote: post #294
marhee wrote: post #230

Earlier quoted context omitted.

Worst case scenario you service is not available for a couple of hours. In 99% of business, customers are totally okay with that (if it's just not every week). IRL shops are also occasionally closed due to incidents; heck even ATMs and banks don't work 100% of the time. And that's the worst case: because your setup is so simple, restoring a backup or even doing a full setup of a new machine is quite easy. Just make s…

Depending on your product, this could mean tens of thousands to millions of dollars worth of revenue loss. I don't really see how we've gone backwards here. You could just distribute your workloads using...a queue, and not have this problem, or have to pay for and pay to maintain backup equipment etc.

It could. In those cases, you set up the guardrails to minimize the loss.

In your typical seed, series A, or series B SaaS startup, this is most often not the case. At the same time, these are the companies that fueled the proliferation of microservice-based architectures, often with a single-point of failure in the message queue or in the cluster orchestration. They shifted easy-to-fix problems into hard-to-fix problems.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#333

Post by immibis »

peoplefromibiza wrote: post #194
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> "You're not Google. Your company will never be Google." I'm not sure people realize this now more than then. I was there back then and we surely knew we would never be Google hence we didn't need to "scale" the same way they did. Nowadays every project I start begins with a meeting where is presented a document describing the architecture we are going to implement, using AWS of course, because "auto-scale" right?,…

AWS overcharges 100x for bandwidth, and CloudFront's free tier has 10x more bandwidth in it.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#334

Post by koliber »

k8sToGo wrote: post #124
steve1977 wrote: post #115

Earlier quoted context omitted.

That‘s nice, but the question is if you (i.e. your company) needs this ecosystem.

Need no. But it’s nice especially since I have the know how. Creating a new deployment is just super fast. Developers can deploy their own apps etc. And then there is Helm. If we only decide by “need” then most of the time we also wouldn’t need object oriented programming.

Lot's of things are nice to have but are expensive.

I'd love to have a private pool in my backyard. I don't, even though it's nice to have, because it is too expensive.

We intuitively make cost-benefit choices in our private lives. When it comes to deciding the same things at work, our reasoning often goes haywire.

Sometimes we need an expensive thing to solve a real problem.

Your point about object-oriented programming makes sense. Sometimes, a bash script suffices, and the person who decides to implement that same functionality in Java is just wasting resources.

All of these solutions have a place where they make sense. When they are blindly applied because they're a fad they generate a lot of costs.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#335

Post by k8sToGo »

koliber wrote: post #334
k8sToGo wrote: post #124

Earlier quoted context omitted.

Need no. But it’s nice especially since I have the know how. Creating a new deployment is just super fast. Developers can deploy their own apps etc. And then there is Helm. If we only decide by “need” then most of the time we also wouldn’t need object oriented programming.

Lot's of things are nice to have but are expensive. I'd love to have a private pool in my backyard. I don't, even though it's nice to have, because it is too expensive. We intuitively make cost-benefit choices in our private lives. When it comes to deciding the same things at work, our reasoning often goes haywire. Sometimes we need an expensive thing to solve a real problem. Your point about object-oriented programm…

That’s true. Using k8s to host a static website would be silly.

Generally I only use it when I see a compelling case for it and the introduced complexity takes away complexity from somewhere else.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#336

Post by masfoobar »

I honestly cannot comment on RabbitMQ or Kafka. I have not used them. I have also not used Redis other than learning.

However, a few years ago, I did use MSMQ (Microsoft Message Queue) and worked very well. However, at the time, I wanted something that didn't limit me to Windows.

In the end, I ended up learning ZeroMQ. Once I understood their "patterns" I created my own Broker software.

Originally, all requests (messages) sent to the Broker were being stored in files. Eventually, I moved over to Sqlite. As the broker is designed to process one thing at a time, was not worried multiple requests were going to sqlite. So now my Broker requires little dependencies.. except ZeroMQ and Sqlite.

(I did not need to worry about async processing as they get passed to the workers)

So, a broker is communicated with a client and a worker/consumer.

- client can ask for state (health of a queue, etc)

- client can send a message (to a queue, etc)

The worker communicates with the broker

- please connect me to this queue

- is there anything in this queue for me?

- here is the result of this message (success or failure)

etc.

I also made use of the pub-sub pattern. I made a GUI app that subscribes to these queues and feeds updates. If there are problems (failures) you can see them, here. I leave it to staff to re-send the message. If its already sent, maybe the subscriber missed that packet. pub-sub is not reliable, afterall.. but works 99% of time.

Overall it has been a great system for my needs -- again, it is lightweight, fast, and hardly costs anything. I do not need a beefy machine, either.

Honestly, I swear by this application (broker) i made. Now, am I comparing it to RabbitMQ or Kafka? No! I am sure these products are very good at what they do. However, especially for smaller companies I work for, this software has saved them a few pennies.

In all I found "Distributed computing" to be rewarding, and ZeroMQ+Sqlite have been a nice combination for my broker.

I have been experimenting with nanomsg-NG as a replacement for ZeroMQ but I just haven't spent proper time of it due to other commitments.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#337

Post by teleforce »

Dalewyn wrote: post #161
ithkuil wrote: post #156

Earlier quoted context omitted.

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

It's kind of mind boggling just how powerful mundane desktop computers have gotten, let alone server hardware. Think about it: That 20 core CPU (eg: i7 14700K) you can buy for just a couple hundred dollars today would have been supercomputer hardware costing tens or hundreds of thousands of dollars just a decade ago.

The RAM also get much larger and cheaper, and it is now possible to have several terabyte (TB) of RAM memory (not storage), in a single PC or workstation. This i7 14700K can support 192 GB RAM but other lower end Xeon CPU W for workstation for example w3-2423 costing around USD350 can support 2 TB RAM albeit only 6-core [1]. But then with not so much more extra budgets you can scale the machine to your heart's content [2].

[1] Intel Xeon w3-2423 Processor 15M Cache, 2.10 GHz:

https://www.intel.com/content/www/us/en/products/sku/233484/...

[2] Intel Launches Xeon W-3400 and W-2400 Processors For Workstations: Up to 56 Cores and 112 PCIe 5.0 Lanes:

https://www.anandtech.com/show/18741/intel-launches-xeon-w-3...

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#338

Post by phyrex »

w10-1 wrote: post #302
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> You're not Google. Your company will never be Google True, but the CTO comes from twitter/meta/google/some open-source big-data project, the director loves databases, etc. So we have 40-100 people managing queues with events driven from database journals. Everyone sees how and why it evolved that way. No one has the skill or political capital to change it. And we spend most of our time on maintenance tasked as "upg…

Meta, where they run a PHP monolith with mysql? ;)

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#339

Post by bamboozled »

sturmdev wrote: post #308
bamboozled wrote: post #294

Earlier quoted context omitted.

Depending on your product, this could mean tens of thousands to millions of dollars worth of revenue loss. I don't really see how we've gone backwards here. You could just distribute your workloads using...a queue, and not have this problem, or have to pay for and pay to maintain backup equipment etc.

From the original post: “Your business is not Google and will never be Google” From the post directly above: “Most businesses…” The thread above is specifically discussing business which won’t lose a significant amount of money if they go down for a few minutes. They also postulate that most businesses fall into this category, which I’m inclined to agree with.

I understand it in practice but I also think it's weird to be working on something that isn't aiming to grow, maybe not to good scale but building systems which are "distributable" from and early stage seems wise to me.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#340

Post by MegaSpaceHamlet »

I kind of stumbled into it. I have a server that's processing a lot of incoming data from field devices that expect quick 200 responses. The processing I was required to do on the data was pretty expensive time-wise, mainly because I have to make multiple calls to a third-party API that isn't highly performant. In order to keep everything stable, I had to delegate the data processing to a separate process via a message broker (Redis, with the Bull npm package as an abstraction layer to handle the message-passing), and I have no regrets. This pattern was suggested in the NestJS documentation, the framework I am using. After I realized the power of this pattern (especially because of my heavy leaning on mentioned third-party API), I started using it in other areas of my application as well, and I find to be a helpful pattern. As far as maintenance goes, I just have Heroku take care of my Redis instance. I can easily upgrade my specs with a simple CLI command. There was a slight learning curve in the beginning, but I got the hang of it pretty quickly, and it's been easy to reason about since.
Post reply on HN