Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PgQue: Zero-Bloat Postgres Queue

github.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: PgQue: Zero-Bloat Postgres Queue

#31

Post by carefree-bob »

ozgrakkurt wrote: post #30
ofrzeta wrote: post #25

Earlier quoted context omitted.

Why trust humans? What do you think about this (honest question, because I think this is quite a bit different to what is understood as vibe coding) https://news.ycombinator.com/item?id=47589856 (TJ Green implementing a new PG extension with the help of Claude Code)

I don't trust humans so much as I trust their reputation as a group. I don't know who TJ Green is, even if they are previously working on a database it would take a lot of time for any new product to be trusted. For example I would trust LLVM but I don't trust Mojo which is headed by the same person. Putting LLMs in the equation, you would also need to trust that LLMs do not create hidden garbage that will rot the co…

curious, what about Mojo makes you not trust it?

Re: PgQue: Zero-Bloat Postgres Queue

#32

Post by ozgrakkurt »

carefree-bob wrote: post #31
ozgrakkurt wrote: post #30

Earlier quoted context omitted.

I don't trust humans so much as I trust their reputation as a group. I don't know who TJ Green is, even if they are previously working on a database it would take a lot of time for any new product to be trusted. For example I would trust LLVM but I don't trust Mojo which is headed by the same person. Putting LLMs in the equation, you would also need to trust that LLMs do not create hidden garbage that will rot the co…

curious, what about Mojo makes you not trust it?

I used both rust and zig when they were new and I found myself focusing too much on the language instead of what I wanted to do with it.

Zig doing breaking changes was especially frustrating for some time.

Also all the things that would be missing from the language. Sometimes I realized I need that feature some time into the development and then the compiler doesn't have it.

Mojo being also a new language, I would think just writing code in C or C++ would be more stable and useful for me.

Re: PgQue: Zero-Bloat Postgres Queue

#33

Post by hardwaresofton »

samokhvalov wrote: post #7
saberd wrote: post #3

I don't understand the latency graph. It says it has 0.25ms consumer latency. Then in the latency tradeof section it says end to end latency is between 1-2 seconds. Is this under heavy load or always? How does this compare to pgmq end to end latency?

(PgQue author here) I didn't understand nuances in the beginning myself We have 3 kinds of latencies when dealing with event messages: 1. producer latency – how long does it take to insert an event message? 2. subscriber latency – how long does it take to get a message? (or a batch of all new messages, like in this case) 3. end-to-end event delivery time – how long does it take for a message to go from producer to co…

Not the original commenter but this is an excellent answer, thanks for the clear explanation — this is what people continue to come to HN for IMO.

Also thanks for all the podcasts and content, always a joy to watch.

Re: PgQue: Zero-Bloat Postgres Queue

#35

Post by victorbjorklund »

what wrote: post #16
pierrekin wrote: post #4

Earlier quoted context omitted.

In Postgres land bloat refers to dead tuples that are left in place during certain operations and need to be vacuumed later. It’s challenging to write a queue that doesn’t create bloat, hence why this project is citing it as a feature.

Can’t you just partition the table by time (or whatever) and drop old partitions and not worry about vacuuming? Why do you need to keep around completed jobs forever?

What about old failed jobs? You might wanna keep them around? And maybe you have retries that have a backoff.

Re: PgQue: Zero-Bloat Postgres Queue

#36

Post by j45 »

mind-blight wrote: post #11

The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc. Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hamm…

"The vacuum pressure is real. "

Felt like llm for a second.

Re: PgQue: Zero-Bloat Postgres Queue

#37

Post by CharlieDigital »

mind-blight wrote: post #11

The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc. Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hamm…

    > Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hammering the DB
Design question here (not familiar enough with this approach with Pg)

Would an alternative be to have a small pool of pollers that would "distribute" the records to a later pool of workers instead of having workers directly poll?

Re: PgQue: Zero-Bloat Postgres Queue

#38

Post by mike_hearn »

andrewstuart wrote: post #14

Postgres is not the only database that does queues. Any database that supports SKIP LOCKED is fine including MySQL, MSSQL, Oracle etc. Even SQLite makes a fine queue not via skip locked but because writes are atomic.

SKIP LOCKED isn't quite enough to get a real MQ product. Oracle has a full blown MQ product inside it transactional with other data.

In particular you need the ability to wait for messages to appear in a queue without polling, and for a proper MQ you need things like message priority, exception queues, multi-queue listening, good scalability etc.

Re: PgQue: Zero-Bloat Postgres Queue

#39

Post by samokhvalov »

hardwaresofton wrote: post #33
samokhvalov wrote: post #7

Earlier quoted context omitted.

(PgQue author here) I didn't understand nuances in the beginning myself We have 3 kinds of latencies when dealing with event messages: 1. producer latency – how long does it take to insert an event message? 2. subscriber latency – how long does it take to get a message? (or a batch of all new messages, like in this case) 3. end-to-end event delivery time – how long does it take for a message to go from producer to co…

Not the original commenter but this is an excellent answer, thanks for the clear explanation — this is what people continue to come to HN for IMO. Also thanks for all the podcasts and content, always a joy to watch.

thank you!

Re: PgQue: Zero-Bloat Postgres Queue

#40

Post by carefree-bob »

ozgrakkurt wrote: post #32
carefree-bob wrote: post #31

Earlier quoted context omitted.

curious, what about Mojo makes you not trust it?

I used both rust and zig when they were new and I found myself focusing too much on the language instead of what I wanted to do with it. Zig doing breaking changes was especially frustrating for some time. Also all the things that would be missing from the language. Sometimes I realized I need that feature some time into the development and then the compiler doesn't have it. Mojo being also a new language, I would th…

Yeah, that makes sense. It's always tempting to focus on tools, whether tweaking the OS, or learning new languages, or new frameworks, editors, etc.
Post reply on HN