[flagged]
Ask HN: Why do message queue-based architectures seem less popular now?
351–360 of 376 posts
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#352Last year I left a job that was about to give me the maintenance of a poorly implemented kafka system. I'm so glad I left before I really had to work with that system.
Since then I've been reading about async and await in the newer versions of javascript and it really threw me for a loop. I needed this to slow down some executing code but as I worked through my problems I realized "my god! this is exactly what we could have used for pub/sub at my last job".
We could have replaced a kafka system as well an enterprise workflow system with javascript and the async/await paradigm. Some of these systems cost millions per year to license and administer.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#353I think probably for garbage collection reasons.
At my [place of work] we have built a simple event system in top of lambda functions, sqs, s3, eventbridge etc to ingest and add metadata to events before sending them on to various consumers.
We replaced an older kafka system that did lots of transformations to the data making it impossible to source the origin of a field at the consumer level; the newer system uses an extremely KISS approach - collate related data without transformation, add metadata and tags for consumers to use as a heads up and then leave it at that.
I agree that most regular stuff should just be http (or whatever) microservices as the garbage collection is free; requests, sockets, etc time out and then there's no rubbish left over. In an event based system if you have issues then suddenly you might have dozens of queues filled with garbage that requires cleanup.
There are definitely pros to event based, the whole idea of "replaying" etc is cool but like...I've never felt the need to do that...ever.
The event volume that we do process is quite low though, maybe a couple hundred k messages a day.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#354Earlier quoted context omitted.
Worst case scenario you service is not available for a couple of hours. In 99% of business, customers are totally okay with that (if it's just not every week). IRL shops are also occasionally closed due to incidents; heck even ATMs and banks don't work 100% of the time. And that's the worst case: because your setup is so simple, restoring a backup or even doing a full setup of a new machine is quite easy. Just make s…
Depending on your product, this could mean tens of thousands to millions of dollars worth of revenue loss. I don't really see how we've gone backwards here. You could just distribute your workloads using...a queue, and not have this problem, or have to pay for and pay to maintain backup equipment etc.
Hellishly and endlessly optimising for profit is how we've gotten the world into its current state, lmao.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#355Maybe one of the reason why it become unpopular is the additional code you have to implement with asynchronous processing in a separate system and tracking in case of errors during processing in the target system.
Its easier and faster to make a Webservice request where you get an instant result you can handle directly in the source System.
Mostly the queue is implemented in the source System where you can monitor and see the processing status in Realtime without delays.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#356I think this: "* The technology just got mature enough that it's not exciting to write about, but it's still really widely used." Messaging-based architecture is very popular
Yep. It’s a really nice architecture for lots of use cases where it fits.
Every new idea goes through the same cycle of overuse until it finds its niches.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#357It's boring tech. They're quietly being used everywhere in big companies doing the heavy lifting.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#358Corollary Question: what do people with prior experience scaling Elixir/Phoenix think about scaling with it?
I've read very strong reports favorable of it. For instance, it also can be scaled using progressively better hardware (like better CPU or more RAM), or horizontally scaling too. that Also that with a database on the same network, there won't be much need for a cache. Presumably, ability to throw CPU and RAM would lessen some needs for queue too.
At the same time, I don't notice much Elixir usage in practice and it has remained a small community.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#359Corollary Question: what do people with prior experience scaling Elixir/Phoenix think about scaling with it? I've read very strong reports favorable of it. For instance, it also can be scaled using progressively better hardware (like better CPU or more RAM), or horizontally scaling too. that Also that with a database on the same network, there won't be much need for a cache. Presumably, ability to throw CPU and RAM w…
Its fantastic to scale with.
I had written a reply to this thread days ago and decided not to post it.
But its amazing that you can do away with all the message queue infra since its essentially baked in.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#360Corollary Question: what do people with prior experience scaling Elixir/Phoenix think about scaling with it? I've read very strong reports favorable of it. For instance, it also can be scaled using progressively better hardware (like better CPU or more RAM), or horizontally scaling too. that Also that with a database on the same network, there won't be much need for a cache. Presumably, ability to throw CPU and RAM w…
Its fantastic to scale with. I had written a reply to this thread days ago and decided not to post it. But its amazing that you can do away with all the message queue infra since its essentially baked in.
How good is the vertical scaling with modern hardware?
Can you really just keep upgrading the hardware and essentially just scale a rails app?