Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's your go-to message queue in 2025?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Ask HN: What's your go-to message queue in 2025?

#71

Post by mdaniel »

mdaniel wrote: post #70
mattbrewsbytes wrote: post #61

Google PubSub is what we use as our message queue, mostly for communicating change data capture via messages to other internal systems. Its typically being consumed by some job system polling on an interval and then doing CRUD to sync changes. Its not very complex and feels like we're running a lot of compute resources to just sync data between systems. Admittedly there isn't good separation of concerns so there is o…

I love hearing "watch out for" stories, because I feel that allows me to be extra vigilant about some aspect when running my own PoC That said, my suspicion about any such aggregation project like that is that context is everything and trying to capture "this sucks" for all the input criteria which produced that outcome is going to be a wall of text that few will write and even fewer will read (ahem, LLM "tl;dr it fo…

Replying to myself so it can be downvoted separately, but in this mythical new AI gonna take our jobs world, who exactly is supposed to carry out the spikes required to know if technology X is a good fit for problem Y in company Z's culture? Vibe Kakfa into place and yolo?

Re: Ask HN: What's your go-to message queue in 2025?

#72

Post by Spivak »

padjo wrote: post #37
lawn wrote: post #30

Earlier quoted context omitted.

I'm curious on how people use Postgres as a message queue. Do you rely on libraries or do you run a custom implementation?

You can go an awfully long way with just SELECT … FOR UPDATE … SKIP LOCKED

I've never found a satisfying way to not hold the lock for the full duration of the task that is resilient to workers potentially dying. And postgres isn't happy holding a bunch of locks like that. You end up having to register and track workers with health checks and a cleanup job to prune old workers so you can give jobs exclusivity for a time.

Re: Ask HN: What's your go-to message queue in 2025?

#73

Post by user928347483 »

Spivak wrote: post #72
padjo wrote: post #37

Earlier quoted context omitted.

You can go an awfully long way with just SELECT … FOR UPDATE … SKIP LOCKED

I've never found a satisfying way to not hold the lock for the full duration of the task that is resilient to workers potentially dying. And postgres isn't happy holding a bunch of locks like that. You end up having to register and track workers with health checks and a cleanup job to prune old workers so you can give jobs exclusivity for a time.

If you go that route you could spawn a parent process , the work is done in a child process ?

Re: Ask HN: What's your go-to message queue in 2025?

#74

Post by wordofx »

Spivak wrote: post #72
padjo wrote: post #37

Earlier quoted context omitted.

You can go an awfully long way with just SELECT … FOR UPDATE … SKIP LOCKED

I've never found a satisfying way to not hold the lock for the full duration of the task that is resilient to workers potentially dying. And postgres isn't happy holding a bunch of locks like that. You end up having to register and track workers with health checks and a cleanup job to prune old workers so you can give jobs exclusivity for a time.

I use a visibility timeout.

So when selecting a message that isn’t started. It also looks for in progress ones that have been going longer than the timeout.

The update sets status, start time, and attempt counter.

If attempt counter equals 3 when the update happens, it sets the message to failed. The return looks at the stats sees failed and raises a notification.

Then if it’s a fix like correcting data or something I just reset the state to have it reprocess.

Never needed to track workers or cleanup jobs etc.

Re: Ask HN: What's your go-to message queue in 2025?

#75

Post by j45 »

lawn wrote: post #30
wordofx wrote: post #28

Postgres. Doing ~ 70k messages/second average. Nothing huge but don’t need anything dedicated yet.

I'm curious on how people use Postgres as a message queue. Do you rely on libraries or do you run a custom implementation?

Built right in using a group of pg functions, or also with a library, or also with a python based tool that happens to use pg for the queue.

Re: Ask HN: What's your go-to message queue in 2025?

#76

Post by j45 »

smittywerben wrote: post #63

Kafka is a write-ahead log, not a queue per se. It handles transactions to the disk. Not across the network. RabbitMQ is neat out of the box. But I went with ZeroMQ at the time. ZeroMQ is cool but during current year I'd only use it to learn from their excellent documentation. Coming from Python, it taught me about Berkeley sockets and the process of building cross-language messaging patterns. After a few projects, i…

It’s less about DB people or not.

Many of us have used both sides and settled on one area to start.

Kafka, et al are amazing. Also almost always overkill in the first x months or years.

It’s not too much of a stretch to model your queue first in something like Postgres, which oddly offers things a little beyond a traditional rbdms, and when the model implementation in the domain reveals itself… it can shine a nice light in the direction of a Kafka, etc.

Re: Ask HN: What's your go-to message queue in 2025?

#77

Post by j45 »

adamcharnock wrote: post #51

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice. In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out. In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. Th…

While someone’s use case would have to be verified, the below is to show that there are streaming options in Postgres.

Would be interesting to get your take on queues vs streams on the below.

I consider myself a little late to the Postgres party after time with other nosql and rbdms, but it seems more and more an ok place to consider beginning from.

For Streaming…

Supabase has some Kafka stream type examples that covers change data capture: https://supabase.com/blog/postgres-wal-logical-replication

Tables can also do some amount of stream like behaviour with visibility and timeout behaviours:

pg-boss — durable job queues with visibility timeouts and retries.

Zilla — supports Postgres as a source using CDC to act as a stream. • ElectricSQL — uses Postgres replication and CRDTs for reactive sync (great for frontend state as a stream

Streaming inside Postgres also has some attention from

Postgres as Event Store https://eventmodeling.org. This can combine event sourcing with Postgres for stream modeling.

pgmq — from Tempo - this is a minimal message queue built on Postgres using append-only design.. Effectively works as a persistent stream with ordered delivery

Re: Ask HN: What's your go-to message queue in 2025?

#78

Post by adamcharnock »

vlvdus wrote: post #55
adamcharnock wrote: post #51

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice. In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out. In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. Th…

What makes Postgres (or any decent relational DB) fall down in this case?

It is simply that I’m unaware of a streams implementation for postgresql. Although another comment is mentioning them, so I’ll read that in some more detail shortly.

I’ve always felt that streams should be implementable via stored procedures, and that it would be a fun project. I’ve just never quite had the driving force to do it.

Re: Ask HN: What's your go-to message queue in 2025?

#79

Post by adamcharnock »

j45 wrote: post #77
adamcharnock wrote: post #51

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice. In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out. In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. Th…

While someone’s use case would have to be verified, the below is to show that there are streaming options in Postgres. Would be interesting to get your take on queues vs streams on the below. I consider myself a little late to the Postgres party after time with other nosql and rbdms, but it seems more and more an ok place to consider beginning from. For Streaming… Supabase has some Kafka stream type examples that cov…

I suspect this comment is LLM generated. There is a 404-ing URL, discussion of queues, and some discussion of Postgres CDC which I believe is Postgres logical replication. Neither of which are a streams implementation on Postgres.

Re: Ask HN: What's your go-to message queue in 2025?

#80

Post by nick0garvey »

Spivak wrote: post #72
padjo wrote: post #37

Earlier quoted context omitted.

You can go an awfully long way with just SELECT … FOR UPDATE … SKIP LOCKED

I've never found a satisfying way to not hold the lock for the full duration of the task that is resilient to workers potentially dying. And postgres isn't happy holding a bunch of locks like that. You end up having to register and track workers with health checks and a cleanup job to prune old workers so you can give jobs exclusivity for a time.

Hold the lock and write a row with timestamp at the time you read.

That row indicates you are the one processing the data and no one else should. When reading, abort the read if someone else wrote that row first.

When you are finished processing, hold the lock and update the row you added before to indicate processing is complete.

The timestamp can be used to timeout the request.

Post reply on HN