Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do message queue-based architectures seem less popular now?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 376 posts

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#251

Post by _xiaz »

peoplefromibiza wrote: post #194
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> "You're not Google. Your company will never be Google." I'm not sure people realize this now more than then. I was there back then and we surely knew we would never be Google hence we didn't need to "scale" the same way they did. Nowadays every project I start begins with a meeting where is presented a document describing the architecture we are going to implement, using AWS of course, because "auto-scale" right?,…

In defense of CDNs they're also pretty neat for cutting down latency, which benefits even the first customer.

Of course that only helps if you don't end up shoving dozens of MBs in Javascript/JSON over the wire.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#252

Post by Copenjin »

d0mine wrote: post #227
Copenjin wrote: post #123

I really hope that people are slowly starting to understand that using kafka and turning it in a single point of failure (yes, it fails) of your architecture is not a good idea. This pattern has it's uses, but if you are using it everywhere, every time you have some sort of notifications because "it's easy" or whatever, you are likely doing it wrong and you will understand this at some point and it will not be pleasa…

It is impressive if kafka is the weakest link in your system (99.99% is achievable and even 99.999% is not impossible with the architecture)

I've yet to see the numbers you list live, but in addition to pure uptime, I've seen a lot of instances where it has been used as the default communication channel instead of proper RPCs, with devastating effects on the long run from the point of view of issues, debugging, predictability, etc...

It's being overused where it has no reason to be, good and simple on paper, awful in the way many are using it. Most of the times, you don't need it.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#253

Post by throwaway7ahgb »

alfiedotwtf wrote: post #180
ithkuil wrote: post #156

Earlier quoted context omitted.

We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster

This! You NEEDED to scale horizontally because machines were just doing too much. I remember when our Apache boxes couldn’t even cope doing SSL so we had a hardware box doing it on ingress!

I used to administrate a small fleet of sun sparc hosts with SSL accelerators. They were so much money $$$$.

I proposed dumping them all for a smaller set of x86 hosts running linux, it took 2-3 years before the old admins believed in the performance and cost savings. They refused to believe it would even work.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#254

Post by throwaway7ahgb »

peoplefromibiza wrote: post #194
hn_throwaway_99 wrote: post #102

I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…

> "You're not Google. Your company will never be Google." I'm not sure people realize this now more than then. I was there back then and we surely knew we would never be Google hence we didn't need to "scale" the same way they did. Nowadays every project I start begins with a meeting where is presented a document describing the architecture we are going to implement, using AWS of course, because "auto-scale" right?,…

5 rps aside, the more data you can push to the edge (your customer) the cheaper it will be and the better performance for you customer.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#255

Post by jonahbenton »

I see a fair amount of Kafka, while most other platforms have diminished. I think that is because people treat Kafka like a database/system of record. A queue is not a system of record.

A lot of the difficulty in modeling a complex system has to do with deciding what is durable state vs what is transient state.

Almost all state should be durable and/but durability is more expensive upfront. So people make tradeoffs to model a transition as transient and put in a queue. One or two or three years in, that is almost always a regretted decision.

Message queues that are not databases/systems of record wind up glossing this durable/transient state problem, and then when you have also this unique piece of infrastructure to support, this is a now you have two problems moment.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#256

Post by rastignack »

throwaway7ahgb wrote: post #253
alfiedotwtf wrote: post #180

Earlier quoted context omitted.

This! You NEEDED to scale horizontally because machines were just doing too much. I remember when our Apache boxes couldn’t even cope doing SSL so we had a hardware box doing it on ingress!

I used to administrate a small fleet of sun sparc hosts with SSL accelerators. They were so much money $$$$. I proposed dumping them all for a smaller set of x86 hosts running linux, it took 2-3 years before the old admins believed in the performance and cost savings. They refused to believe it would even work.

I have the same memories, trying to convince people to dump slow as hell sparc processors for database workloads in favor of X86 machines costing a 10th of the price.

To this day I still argue with ex Solaris sysadmins.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#257

Post by captainbland »

I think it's both. They're boring if you need them, because you probably started implementing these things over a decade ago. They're boring if you don't need it because you don't need it - e.g. maybe you're a small team able to work on a single monolithic codebase effectively.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#258

Post by ElevenLathe »

_xiaz wrote: post #251
peoplefromibiza wrote: post #194

Earlier quoted context omitted.

> "You're not Google. Your company will never be Google." I'm not sure people realize this now more than then. I was there back then and we surely knew we would never be Google hence we didn't need to "scale" the same way they did. Nowadays every project I start begins with a meeting where is presented a document describing the architecture we are going to implement, using AWS of course, because "auto-scale" right?,…

In defense of CDNs they're also pretty neat for cutting down latency, which benefits even the first customer. Of course that only helps if you don't end up shoving dozens of MBs in Javascript/JSON over the wire.

Putting your app behind a CDN also gives you some cheap defense against (most, casual) DDoS.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#259

Post by tbcj »

RangerScience wrote: post #108
appplication wrote: post #93

Earlier quoted context omitted.

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…

Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things…

If you’re already in the Kubernetes system, Argo Workflows has either capabilities designed around what you are describing or can be built using the templates supported (container, script, resource). If you’re not on Kubernetes, then Argo Workflows is not worth it on its own because it does demand expertise there to wield it effectively.

Someone suggested Temporal below and that’s a good suggestion too if you’re fine with a managed service.

Re: Ask HN: Why do message queue-based architectures seem less popular now?

#260

Post by [deleted] »

1oooqooq wrote: post #132
willturman wrote: post #95

I implemented RabbitMQ based messaging queues as a mechanism to coordinate execution among discrete components of a handful of ambitious laboratory automation systems ~4-8 years ago. Given a recent opportunity to rethink messaging based architectures, I chose the simplicity and flexibility of Redis to implement stack and queue based data-structures accessible across distributed nodes. With even a handful of nodes, it…

sounds like you're about to reinvent a queueing system on top of redis. in a very painful way.

[deleted]
Post reply on HN