Ask HN: Why do message queue-based architectures seem less popular now?

> You're not Google. Your company will never be Google

True, but the CTO comes from twitter/meta/google/some open-source big-data project, the director loves databases, etc.

So we have 40-100 people managing queues with events driven from database journals.

Everyone sees how and why it evolved that way. No one has the skill or political capital to change it. And we spend most of our time on maintenance tasked as "upgrades", in a culture with "a strong role for devops".

If your product going down for an hour will lead to the loss of millions of dollars, then you should absolutely be investing a lot of money in expensive distributed and redundant solutions. That's appropriate in that case.

The point here is that 99% of companies are not in that scenario, so they should not emulate the very expensive distributed architectures used by Google and a few other companies that ARE in that scenario.

For almost all companies on the smaller side, the correct move is to take the occasional downtime, because the tiny revenue loss will be much smaller than the large and ongoing costs of building and maintaining a complex distributed system.

Thanks no. I rather wait 10ms for a rebuild and 1s on gdb init, than 25m for a monolith rebuild, and 2m on gdb init.

Seperate processes, yes.

From the original post: “Your business is not Google and will never be Google”

From the post directly above: “Most businesses…”

The thread above is specifically discussing business which won’t lose a significant amount of money if they go down for a few minutes. They also postulate that most businesses fall into this category, which I’m inclined to agree with.

"If people have experience designing or implementing greenfield systems based on message queues, I'd be curious to hear about it."

Ok, you asked.

It was early Summer 1997 and I had just graduated college with a computer information systems degree. I was employed at the time as a truck driver doing deliveries to Red Lobster and my father was not happy and offered me double my current pay and guaranteed 40 hours per week to return to work for him as an electrician. I returned to work with my Dad for the Summer but after 14+ years of electrical work with him I decided I needed to get a job with computers. Labor day weekend of 1997 I blasted out over 40 resumes and on Tuesday 9/9/97 I had my first interview at a small payments startup dotcom in Wilmington Delaware, the credit card capital of the world at that time. I was hired on the spot and started that day as employee #5 and was the first software hire for the company yet I had NO IDEA what I was doing. I was tasked with creating a program that could take payments from a TCPIP WAN interface and proxy it back out over serial modems to Visa and batch ACH files to the U.S. FED. This started as a monolith design and we were processing automated clearing house as well as credit cards by late 1997. I would continue in this architectural design and sole software developer support of this critical 100% uptime system for many years. Somewhere around mid 1998 volume started to increase and the monolith design experienced network socket congestion, the volume from the firehose continued to increase and I was the sole guy tasked with solving it. That volume increase came from a little known company at the time, PayPal. Since the 'mafia' was very demanding they knew I was the guy however management isolated me since the 'mafia' demanded extensive daily reporting that only the guy who built the platform could provide. This however took a backseat to the network connection issues which were growing at an increasing rate. I was involved in a lot of technology firsts as a result and herein starts the queue story. After processing payments for Microsoft TechEd too in 1998 I was given an NT Option Pack CD. I was constantly seeking a solution to reduce the network congestion on the monolith and within this option pack was something called "Microsoft Message Queue". I spent several months nonstop of nights and weekends redesigning the entire system using an XML interface API from the ground up while writing individual services that read from an ingress queue and output to and egress queue, this structure is now know as microservices and this design solved all the load problems since it scaled extremely well. This new redesigned system had many personal experience enhancements added such as globally unique identifiers as well as the API being fully extensible but the greatest unseen win was the ability to 100% recreate software bugs since all message passing was recorded. Paypal ended up leaving us in ?2003? for JPMorgan after the majority holder refused to sell to the 'mafia'. Some years later I was informed by several management executives that Paypal also offered to exclusively hire only me but I was of course never informed of that.

I have many a business horror story however in over a decade of using MSMQ at my first payments company, 1998-2010, I only had one data corruption event in the queue broker which required me to reverse engineer the MSMQ binary and thus file format to recover live payments records. This corruption was linked to bad ECC memory on one of those beige 1U Compaq servers that many here likely recall.

This story may reveal my age but while my ride has been exciting it isn't over yet as so many opportunities still exist. A rolling rock gathers no moss!

Stay Healthy!