[dead]
Ask HN: Why do message queue-based architectures seem less popular now?
31–40 of 376 posts
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#32Queueing systems haven't disappeared as thery're an important part of distributed systems.
If you need to distribute work/data asynchronously to multiple workers, you're gonna use a queuing system.
Although queuing systems can be implemented on top of a database, message queues like RabbitMQ / ZeroMQ are doing a fine job. I use RabbitMQ all the time, precisely because i need to transfer data between systems and i have multiple workers working asynchronously on that data.
I guess these architectures might be less popular, or less talked about, because monoliths and simple webapps are more talked about than complex systems ?
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#33My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services. These days all those pe…
I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should presumably happen asynchronously, and possibly passing through the hands of more than one SaaS product en route. So given what seems to be the large volume of async messages, in varying/spiking volumes, possibly interacting with 3rd party services which will sometimes experience outages ... I gotta expect that a bunch of these systems are solving "actual business problems" of separating out work that can be done later/elsewhere, can fail and be retried without causing disruptions, etc in order to ensure the work that must happen immediately is protected.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#34In our case, we managed to solve most of the use cases with less specialized tools.
I still think queues are great, but most of the time I can just run my queues using language constructs (like Channels) communicating between threads. If I need communication between machines, I can usually do that with Postgres or even s3. If you're writing constantly but only reading occasionally, you don't need to consume every message – you can select the last five minutes of rows from a table.
I've also seen a general trend in my line of work (data engineering) to try and write workloads that only interact with external services at the beginning or end, and do the rest within a node. That makes a lot of sense when you're basically doing a data -> data transformation, and is easier to test.
There are still cases where we need the full power of Kinesis, but it's just a lot less common than we thought it would be 10 years ago.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#35[deleted]
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#36The hype was cooled down once companies realized "they have more microservices than actual users"
Was never obsessed with "event-driven" distributed systems using message queues.
The major issue is to keep syncing state between services.
Quite for a long time used to get decent results with simple golang and postgres scripts to distribute work between workers on multiple bare metal machines
Took ideas from projects similar to MessageDB
https://redis.io/docs/latest/develop/data-types/streams/
CREATE TABLE IF NOT EXISTS message_store.messages ( global_position bigserial NOT NULL, position bigint NOT NULL, time TIMESTAMP WITHOUT TIME ZONE DEFAULT (now() AT TIME ZONE 'utc') NOT NULL, stream_name text NOT NULL, type text NOT NULL, data jsonb, metadata jsonb, id UUID NOT NULL DEFAULT gen_random_uuid() );
along with Notify https://www.postgresql.org/docs/9.0/sql-notify.html
or polling techinques
Redis Stream here and there worked well too https://redis.io/docs/latest/develop/data-types/streams/
Another possible alternative can be "durable execution" platforms like Temporal, Hatchet, Inngest mentioned on HN many times
- Temporal - https://temporal.io/ - https://github.com/temporalio - https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- Hatchet https://news.ycombinator.com/item?id=39643136
- Inngest https://news.ycombinator.com/item?id=36403014
- Windmill https://news.ycombinator.com/item?id=35920082
Biggest issues I had with workflow platforms - it requires a huge congnitive investments in understanding SDKs, best practices, deciding how properly persist data. Especially, if you optout to host it yourself.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#37My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services. These days all those pe…
I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should…
Bingo - I work on the backend of a medical system and basically anything that interacts with a 3rd party gets put into a queue so our application doesn't choke immediately when one of them has issues. We also have some uses for it within our system.
As far as the question, I was thinking that queues have probably just become a standard aspect of modern distributed systems; it's considered a pretty foundational cloud service for any provider (though we just run RabbitMQ ourselves and it has worked well for us).
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#38Every one of my customers uses message queues either Redis, SQS and rarely Kafka using Celery, Sidekiq, etc. If anything it is boring and works and just part of the default stack everyone uses.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#39I think this: "* The technology just got mature enough that it's not exciting to write about, but it's still really widely used."
Messaging-based architecture is very popular
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#40I never fully understood the need for back end message queues TBH. You can just poll the database or data store every few seconds and process tasks in batches... IMO, the 'real time' aspect was only ever useful for front end use cases for performance reasons since short polling every second with HTTP (with all its headers/overheads) is prohibitively expensive. Also, HTTP long polling introduces some architectural complexity which is not worth it (e.g. sticky sessions are required when you have multiple app servers).
Unfortunately, moving real-time messaging complexity entirely to the back end has been the norm for a very long time. My experience is that, in general, it makes the architecture way more difficult to manage. I've been promoting end-to-end pub/sub as an alternative for over a decade (see https://socketcluster.io/) but, although I've been getting great feedback, this approach has never managed to spread beyond a certain niche. I think it's partly because most devs just don't realize how much complexity is added by micromanaging message queues on the back end and figuring out which message belongs to what client socket instead of letting the clients themselves decide what channels to subscribe to directly from the front end (and the back end only focuses on access control).
I think part of the problem was the separation between front end and back end developer responsibilities. Back end developers like to ignore the front end as much as possible; when it comes to architecture, their thinking rarely extends beyond the API endpoint definition; gains which can be made from better integrating the back end with the front end is 'not their job'. From the perspective of front-end developers, they see anything performance-related or related to 'architectural simplicity' as 'not their job' either... There weren't enough full stack developers with the required insights to push for integration efficiency/simplicity.