Live data from Hacker News

Ask HN: What's your go-to message queue in 2025?

news.ycombinator.com

81–90 of 99 posts

Re: Ask HN: What's your go-to message queue in 2025?

#81

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice. In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out. In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. Th…

There are lots of options to stream data out of Postgres, including:

- https://electric-sql.com (disclaimer: co-founder) - https://feldera.com - https://materialize.com - https://powersync.com - https://sequinstream.com - https://supabase.com/docs/guides/realtime/broadcast - https://zero.rocicorp.dev

Etc.

Re: Ask HN: What's your go-to message queue in 2025?

#82
post #81

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice. In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out. In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. Th…

There are lots of options to stream data out of Postgres, including: - https://electric-sql.com (disclaimer: co-founder) - https://feldera.com - https://materialize.com - https://powersync.com - https://sequinstream.com - https://supabase.com/docs/guides/realtime/broadcast - https://zero.rocicorp.dev Etc.

I think these all relate to streaming data. Not streams in the sense of the data-structure for message passing (a la Kafka, Redis Streams, etc)

Re: Ask HN: What's your go-to message queue in 2025?

#83
1. Never do greenfield. But usually seen systems set up with the cloud "house white" queue. SQS or the Azure queue whatever its called.

2. Nothing. It all worked out.

3. Nowhere. Generally used them for queue-y things.

4. Not done this. Even back in 2000s when queues weren't so well known they'd be a queue-like system. Polling FTP for example!

Re: Ask HN: What's your go-to message queue in 2025?

#84

I prefer pulsar. Elegant modular design and fully open source ecosystem. Performance is at least as good as Kafka. For simpler workload, beanstalkd could be a good fit, either.

Pulsar's feature set is amazing, but it looks like a beast to operate? Especially compared to lighter-weight systems like NATS or Redpanda.

You need both Bookkeeper and Pulsar, which are both stateful, and both require ZooKeeper. (You can apparently configure Bookkeeper to use Etcd, not sure about Pulsar.) So three applications, each of which has several types of processes that probably demand a dedicated operator if running on Kubernetes.

Re: Ask HN: What's your go-to message queue in 2025?

#85
It's important to distinguish between the use cases. Queues, streams, logs, databases, etc. are different kinds of tools you can use, and what the right tool is depends on your semantics.

For example: Message queues are good for work that must be done in strict order where you want to deal with one message at a time. They aren't such a great fit for large batch movement of data, like logs or high volume events, because having a per-message acknowledgement state requires a lot of round trips over the network that simply isn't needed; you want to treat the entire bulk of the flow to carve out big chunks of it, because CPUs wnd networks and disks are more efficient when doing the same operation over large amounts of data in one go.

If you are executing "tasks" (like image processing, ML inference, webhooks), ordering by insertion order might not be the right choice, either. Sometimes you want to coalesce (dedupe by key). Sometimes you want to ensure the processing for a key (e.g. a customer ID) is done in the same process and not randomly distributed over all your workers. Sometimes you want delivery to be strictly sequential, requiring an exclusive worker rather than massively parallel fan-out. And so on.

Where I work, we use a mix of things depending on the application. I am a big fan of NATS. It's not itself a message queue, but its primitives can be combined to handle all sorts of behaviors. Core NATS is more like ephemeral pub/sub, while Jetstream gives you durable, highly available Kafka-like streams.

I like combining queues with database state. Use the queue as an efficient way to order items (like jobs or events) for massively scalable distribution, and use the database to store the current state of things.

For example, imagine you're delivering webhook messages. We first store the message in the database with the state "pending", then write an event to the queue about it. The worker receives the event, double-checks its state is still "pending", then executes it. If delivered, mark as "done" and ack the message. Otherwise, mark as "failed" and create a new queue message to retry. This way, you have durable state in a solid database, and the queue is an efficient way to coordinate the workers. (There's a bit more work here to ensure consistency, but this is the gist of it.)

Core NATS is fantastic as a communication primitive between ephemeral processes. You can use it for RPC, for lightweight broadcasts (e.g. reload config everywhere), even for things like leases or caching or similar. Jetstream is like Kafka but more flexible; for example, each message has a wildcard subject that can be filtered on, so different consumers can very efficiently filter a big, commingled stream by interest. In Jetstream streams, messages have per-consumer ack/nack state in addition to a position, so you're not limited to Kafka's linear "position". Overall, a superb data model, and very easy to manage as infra.

One weak point with NATS is a maximum message size of 10MB. This means that you sometimes have to invent your own chunking if your application needs to send larger payloads. Doing this opens up some cans of worms, so I honestly wouldn't recommend it. For large batch stuff, Redpanda is a better option.

Re: Ask HN: What's your go-to message queue in 2025?

#86
post #39
post #33

The US Federal Reserve uses IBM MQ for the FedNow interbank settlement service that went live last year. Architecture info: https://explore.fednow.org/resources/technical-overview-guid...

Likely implies z/OS is common on both sides. Given the stakes and availability needs not a bad choice.

In the linked doc they say it’s MQ on AIX but I believe it’s interchangeable with Z.

Re: Ask HN: What's your go-to message queue in 2025?

#87

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice. In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out. In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. Th…

Great comment. I'm disappointed that I had to scroll this far down to see someone pointing out that queues and streams ARE NOT THE SAME.

Re: Ask HN: What's your go-to message queue in 2025?

#88
post #81

Earlier quoted context omitted.

There are lots of options to stream data out of Postgres, including: - https://electric-sql.com (disclaimer: co-founder) - https://feldera.com - https://materialize.com - https://powersync.com - https://sequinstream.com - https://supabase.com/docs/guides/realtime/broadcast - https://zero.rocicorp.dev Etc.

I think these all relate to streaming data. Not streams in the sense of the data-structure for message passing (a la Kafka, Redis Streams, etc)

The logical replication of the transaction log is basically a stream of data change events, so the difference between those senses isn’t very big.

Re: Ask HN: What's your go-to message queue in 2025?

#89
post #76

Kafka is a write-ahead log, not a queue per se. It handles transactions to the disk. Not across the network. RabbitMQ is neat out of the box. But I went with ZeroMQ at the time. ZeroMQ is cool but during current year I'd only use it to learn from their excellent documentation. Coming from Python, it taught me about Berkeley sockets and the process of building cross-language messaging patterns. After a few projects, i…

It’s less about DB people or not. Many of us have used both sides and settled on one area to start. Kafka, et al are amazing. Also almost always overkill in the first x months or years. It’s not too much of a stretch to model your queue first in something like Postgres, which oddly offers things a little beyond a traditional rbdms, and when the model implementation in the domain reveals itself… it can shine a nice li…

Sir, your reply is more coherent than mine. I'll give you props for that.

Still, I disagree that Kafka is always overkill.

When god opened a datagram socket on your computer, you needed to have been capturing this data X months ago, but weren't paying attention. You need to build warbot.py and put it into production before you have the chance to deal with cold storage. Kafka is my go-to if you can do this before you run out of disk space.

I frequently append JSON lines to a "data.json" file in Python. Add a socket server, compression, and typed bindings for 20+ languages. Boom. Kafka. Don't oversell it. Need to delete a row? Congratulations, you selected the wrong tool. It's appending JSON lines to a file. Kafka is a write-ahead log, not a queue.

To your point about Postgres, I've found Postgres has fantastic JSONB support and awesome developers who have been very influential in my life and whom I admire. Postgres is my preferred cold storage, which I connect to Kafka. It feels like swimming upstream because RMDBs are traditionally for normalized data, not denormalized JSON lines that make XML look hip again.

If you have a choice in DB, Postgres' JSONB has helped me avoid unnecessary normalization steps. It's good to have options.

ZeroMQ would call this the Titanic pattern and mic drop because the guide has a section on it. That's why I like ZeroMQ.

Edit: Apologies for typos/brevity. I have an ancient phone that only works with 20% of the web and phone apps. There are no apps or LLMs to help this dyslexic soul.

Reference for the Titanic pattern. The guide's author is cynical about me shoving spinning rust in the middle, but it doesn't say no. https://zguide.zeromq.org/docs/chapter4/#Disconnected-Reliab...

Re: Ask HN: What's your go-to message queue in 2025?

#90

Kafka is a write-ahead log, not a queue per se. It handles transactions to the disk. Not across the network. RabbitMQ is neat out of the box. But I went with ZeroMQ at the time. ZeroMQ is cool but during current year I'd only use it to learn from their excellent documentation. Coming from Python, it taught me about Berkeley sockets and the process of building cross-language messaging patterns. After a few projects, i…

I already spent time proofreading it, but I forgot that I couldn't edit it after so much time. Unless the merciful admin lets me roll the 12-sided dice to replace the above cringe with the below piece I carved out of soap.

---

I always check for maintained libraries for my programming languages for any messaging library. Bindings in many languages are consistent across Kafka, ZeroMQ, and NATS.

Kafka is a write-ahead log, not a queue per se. It handles transactions to the disk. The networking is a simple broadcast, not a shared queue. You also can't (canonically, at least) pop/insert/delete rows. It's append-only. It can do basic seeking, like replaying from the start.

ZeroMQ is a good choice for learning from its excellent documentation, and programmers interested in C programming. Probably a good lead into Beej's networking guide. ZeroMQ is the odd one as it has no central broker ("Zero" for zero broker); you copy your favorite broker.py pattern from the ZeroMQ guide.

Dropping anchor to throw in the POSIX standard sockets, the BSD kqueue, the Linux epoll, newer io_uring, and libuv for boring cross-platform asynchronous I/O.

https://zguide.zeromq.org/docs/preface/

https://beej.us/guide/bgnet/html/split/

https://pubs.opengroup.org/onlinepubs/9699919799/functions/p...

https://pubs.opengroup.org/onlinepubs/9699919799/functions/s...

https://man.freebsd.org/cgi/man.cgi?kqueue

https://man7.org/linux/man-pages/man7/epoll.7.html

https://man7.org/linux/man-pages/man7/io_uring.7.html

https://docs.libuv.org/en/v1.x/

Post reply on HN