Ask HN: Why do message queue-based architectures seem less popular now?

Libraries are a great way to manage separation of concerns. Any dependency you add has to be explicit. There's nothing stopping you from adding that dependency but you can't just do it accidentally.

The graph of dependencies between components makes for explicit separation of concerns just like you would have a graph of dependencies between different network services.

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so).

For example, you need some ETL to happen every day. Instead of having your pipeline logic inside an airflow task, you put your logic in a library, where you can test and establish boundaries for this behavior in isolation, and compose this logic portably into any system that can accept your library code. When you need to orchestrate, you just call this function inside an airflow task.

This has a few benefits. You now decouple, to a significant extent, your logic and state transfer from your orchestration. That means if you want to debug your DAG, you don’t need to do it in Airflow. You can take the same series of function calls and run them, for example, sequentially in a notebook and you would achieve the same effect. This also can reveal just how little logic you really need in orchestration.

There are some other tricks to making this work really well, such as reducing dependency injection to primatives only where possible, and focusing on decoupling logic from configuration. Some of this is pretty standard, but I’ve seen teams not have a strong philosophy on this and then struggle with maintaining clean orchestration interfaces.

I implemented RabbitMQ based messaging queues as a mechanism to coordinate execution among discrete components of a handful of ambitious laboratory automation systems ~4-8 years ago.

Given a recent opportunity to rethink messaging based architectures, I chose the simplicity and flexibility of Redis to implement stack and queue based data-structures accessible across distributed nodes.

With even a handful of nodes, it was challenging to coordinate a messaging based system and the overhead of configuring a messaging architecture, essentially developing an ad-hoc messaging schema with each project (typically simple JSON objects), and relatively opaque infrastructure that often required advanced technical support led messaging systems to fall out of favor for me.

Kafka seems to be the current flavor of the day as far as messaging based systems, but I don't think I'll ever support a system that approaches the throughput required to even think about implementing something like Kafka in the laboratory automation space - maybe there's a use case for high-content imaging pipelines?

Right now, I'd probably choose Redis for intra-system communication if absolutely necessary, then something like hitting a Zapier webhook with content in a JSON object to route information to a different platform or software context, but I'm not in a space where I'm handling Terabytes of data or millions of requests a second.

> Order confirmation emails and sms messages are triggered by the order submission, and again usually sent off to a 3rd party bulk email or SMS service. Twilio or Campaign Monitor or Mailchimp will have queues and retry mechanisms, but again the website devs are just firing off an API call to some 3rd party that's dealing with that.

In my case, I need to compile templates for the e-mails to be sent, which is somewhat slow. Even if I have an in memory cache for the compiled templates that can then be quickly filled in with actual data, I don't want to make the first user to request them after a restart/flush wait like 4-7 extra seconds upon clicking on the confirm order button (in addition to any other 3rd party calls, checking payment status etc.).

Ergo, I need to carry the actual e-mail preparation logic (regardless of whether I send it myself, or use a 3rd party service) out into a separate thread. The problem then is that I most likely don't want to create a new thread for every e-mail to be processed, so I need a queue for the requests and one or multiple worker threads. There is functionality for this in the stack I'm using so no big deal, except I can't pass live DB entities across threads, so I also need to serialize the data before passing it off to the queue and deserialize it inside of the queue (or just pass some IDs and do DB calls within the queue).

Essentially I've created a simple queue system in the app code, since processing that data definitely shouldn't make the user wait on it. I can see why some might also go the extra step and opt for something like RabbitMQ, since at the end of the day my queue system is likely to be way more barebones than something that's been around for years. But until a certain point, YAGNI.

IMO, many "microservices" just needed a way to do async processing without needing to hold on to connections, and landed up leveraging msg queues for such use cases. Now this is mostly getting replaced by new orchestration workflows like was step functions/temporal/orkes etc