Consensus in distributed systems is hard and MQs don’t help.
If possible boiled down systems to a transitional db replica set. That is a local minimum in complexity that will serve you for a long time.
Ask HN: Why do message queue-based architectures seem less popular now?
51–60 of 376 posts
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#52I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…
In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…
Microservices necessarily add more complexity and overhead when compared to a monolith. Just the fact that you have to orchestrate N services instead of just pressing run on a single project demonstrates some of the additional complexity.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#53Message queues have moved on past the "Peak of inflated expectations" and past the "trough of disillusinment" into the "slope of enlightenment", perhaps even the "plateau of productivity". https://en.wikipedia.org/wiki/Gartner_hype_cycle
I used zmq to build our application used for testing new hardware. Everything comes in via serial every second so I made a basic front end for that serial bus that sends telemetry out over zmq and waits for any commands coming in using pub/sub. The front end piece can sit forever sending out telemetry no one ever hears or I can hook up a logger, debug terminal, data plotter, or a factory test GUI that runs scripts or all at once. Dealing with com ports on windows is a huge hassle so zmq lets me abstract those annoyances away as a network socket. Other engineers can develop their own applications custom to their needs and they have. Our old application tried to shove all of this functionality into one massive python script along with trying to update a complicated Tk GUI every second with new telem. The thing was buckling under its own weight and would actually screw up the serial data coming in if you were running a heavy script in another thread. I know there are ways to turn a serial port into a network socket but I wanted something that didn't require a server or client to be online for the other to function.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#54Earlier quoted context omitted.
I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should…
One word: scale. The services you mention above do require scale if commercial. OP argues and I somewhat agree that lots of resume driven tech was oversold and overused making things more complicated and expensive than they should have. Once tech gets more mature it’s harder do misuse and it is used where real needs arise.
This is true, but in my opinion badly misunderstood.
There are a huge number of "commercial" things that are hitting several million dollars a month in revenue running Ruby On Rails or WordPress/PHP. You can scale a long long way with "boring technology".
Way too many people think that are "the unicorn" who's user base is going to grow so quickly that then need billion simultaneous user scale right now - instead of realising that before they get even close to that they'll be generating enough revenue to have an engineering team of hundreds who will have rewritten everything two or three times with more suitable architectures already.
If you need a billion users to turn a profit, then whether you admit it or not your business model is "burn VC money until they stop giving it to us or we pivot using Doctorow's Enshittification blog posts as a guide book". That though, is a vanishingly small percentage of all dev work. Most business models have a way too make real profits of thousands or perhaps tens or hundreds of thousands of transactions a month - and they should be rolling in profit to reinvest in future higher scale development well before they run out of fairly pedestrian "boring technology" platforms. Horizontally scalable Java/PHP/Ruby/Node with vertically scaling databases on you cloud provider of choice is a well known and battle tested way to generate real value and cashflow in probably 99% of all businesses.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#55I think this: "* The technology just got mature enough that it's not exciting to write about, but it's still really widely used." Messaging-based architecture is very popular
This is the answer. I'd wager that almost every distributed system that runs at scale uses message queues in some capacity.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#56I've gone though all of these at different scales. What I find these days are: 1. databases have gotten very good where a separate queue infrastructure isn't worth it, 2. databases provide much better observability (you can query the contents). If you really do want a queue, you're still better off using a stream (e.g. Kafka) for multiple producers and/or consumers. Using a database table as a 'transactional outbox' is a poor-man's Kafka which works well enough without the additional infrastructure for most scales.
Redis taking some of the duty as you mentioned and microservices/distributed systems being less fashionable likely also factor into it.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#57I think this: "* The technology just got mature enough that it's not exciting to write about, but it's still really widely used." Messaging-based architecture is very popular
Agreed. It has become a tool just like any other. Just like nobody writes about how they use virtual machines in the cloud anymore.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#58Earlier quoted context omitted.
In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…
Microservices necessarily add more complexity and overhead when compared to a monolith. Just the fact that you have to orchestrate N services instead of just pressing run on a single project demonstrates some of the additional complexity.
Counterpoint: a monolith usually contains a complex init system which allows multiple ways of running the codebase. Microservices can avoid at least that one complexity.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#59The pendulum just swung the other way. Message queues were complicated and don't solve every problem, so rather than think about what the individual needs, the "community" took a 180 and adopted solutions that were overly simplistic and didn't solve every problem. It happens every 5 years or so with a different tech thing. HN is nothing if not constantly chasing any contrarian opinion that seems novel.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#60I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just was…
In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessar…
In my experience, the biggest issue with microservices is that they convert nice errors with a stack trace to network errors. Unless you also invest heavily in observability (usually using expensive tools), running and debugging monoliths generally seems easier