Ask HN: What's your go-to message queue in 2025?

Its not very complex and feels like we're running a lot of compute resources to just sync data between systems. Admittedly there isn't good separation of concerns so there is overlap that requires data syncs.

I've been looking at things like kafka, etc. thinking there might be some magic there that makes us use less compute or makes data syncs a little easier to deal with but wonder what scale of data throughput is a tipping point where a service like that is really needed. If it turns out its just a different service but same timeliness of data sync and similar compute resources I struggle with what benefits might be provided.

I'd love for almost like a levels.fyi style site where people could anonymously report things like this for the tech stacks being used, throughput of data, amount of compute in play, and ratings/comments on their overall solution ("would do again", "don't recommend", "overkill", "resume filler"). It feels much like other areas of technology where a use case comes out of a huge company and RDD (resume driven development) takes hold and now there are people out there doing the equivalent of souping up a 1997 honda accord like its a racecar but its only driving grandma to her appointments.

RabbitMQ is neat out of the box. But I went with ZeroMQ at the time.

ZeroMQ is cool but during current year I'd only use it to learn from their excellent documentation. Coming from Python, it taught me about Berkeley sockets and the process of building cross-language messaging patterns. After a few projects, it's like realizing I didn't need ZeroMQ to begin with I could make my own! If ZeroMQ's Hintjens were still with us I'd still be using it.

It's like the documented incremental process of designing a messaging queue to fit your problem domain, plus a thin wrapper easing some of lower level socket nastiness. At least that's my experience using it over the years. Me talking about it won't do it enough justice.

NATS does the lower level socket wrapper part very nicely. It's a but more modern too. Golang's designed to be like a slightly nicer C syntax, so it would make sense that it's high performance and sturdy. So it's similar to ZeroMQ there.

I'm not sure if either persist to disk out of the box. So either of these are going to be simpler and faster than Kafka.

The DB people are probably trying too hard to cater to the queues. Ideally I'd have normalized the data and modeled the relations such transactions don't lock up the whole table. Then I started questioning why I needed a queue at all when databases (sans SQLite which is fast enough as is) are made for pooling access to a database.

Kafka supports pipelining to a relational database but this part is where you kind of have to be experienced to not footgun and I'm not at that level. I think using it as a queue in that you're short-circuiting it from the relational database pipeline is non-standard for Kafka. I suspect that's where a lot of the Kafka hate is from. I could understand if the distributed transactions part is hell but at that point it's like why'd you skip the database then? Trying to get that free lunch I assume.

I have an alternative. Try inserting everything into a SQLite file. Running into concurrency issues? Use a second SQLite file. Two computers? send it over the network. More issues? Since it's SQL just switch to a real database that will pool the clients. Or switch to five of them. SQL is sorta cool that way. I assume that would avoid the reimplementing half of the JVM to sync across computers where you get Oracle Java showing up to sell you their database halfway into making your galactic scale software or the whatever.

I must be stressed today. Thanks for asking.

I have an idea of a project where even MySql/Maria is too much of admin burden.

On the consumer side the duty cycle drives design. If it’s a steady flow then a polling listener is easy to right size. If the flow is episodic (long periods of idle with unpredictable spikes of high load) one option is to put a alarm on the queue that triggers when it goes from empty to non-empty, and handle that alarm by starting the processing machinery. That avoids the cost of constantly polling during dead time.