Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

351–360 of 369 posts

Re: Choose Postgres queue technology

#351

Post by adamckay »

runeks wrote: post #347
bunderbunder wrote: post #296

Earlier quoted context omitted.

Similar experience here. Multiple times, I've pushed an SQL-based queue a couple orders of magnitude past the scale where others say SQL craps out and a distributed solution is an absolute requirement. And the SQL solution is typically simpler, requires fewer compute resources, and easier to support in production. But, to make it work, you've got to know the database well enough to know that things like SELECT FOR UP…

> Multiple times, I've pushed an SQL-based queue a couple orders of magnitude past the scale where others say SQL craps out and a distributed solution is an absolute requirement. What about availability, though? The distributed solution is also useful to avoid downtime in case of single node failure. Is there an off-the-shelf solution that lets me do that with Postgres? I know the newest version (16) just added activ…

But if you're using Postgres as your queuing system because you're already using it as your core database technology for your app, you've got the same issue. If your single Postgres instance is down then your app is, too, and won't be enqueuing more jobs.

And unless your jobs are trivial then it's highly likely that they interact with your app in some way so it doesn't really matter if your workers are distributed and up, they're not able to complete their work because your app is down because of a single-node Postgres.

Re: Choose Postgres queue technology

#352

Post by dekkers »

runeks wrote: post #347
bunderbunder wrote: post #296

Earlier quoted context omitted.

Similar experience here. Multiple times, I've pushed an SQL-based queue a couple orders of magnitude past the scale where others say SQL craps out and a distributed solution is an absolute requirement. And the SQL solution is typically simpler, requires fewer compute resources, and easier to support in production. But, to make it work, you've got to know the database well enough to know that things like SELECT FOR UP…

> Multiple times, I've pushed an SQL-based queue a couple orders of magnitude past the scale where others say SQL craps out and a distributed solution is an absolute requirement. What about availability, though? The distributed solution is also useful to avoid downtime in case of single node failure. Is there an off-the-shelf solution that lets me do that with Postgres? I know the newest version (16) just added activ…

What the article writes about scalability also applies to availability. Does the queue need 99.999% or 99.9999% uptime? Or is the Service Level Objective actually 99.99%, 99.9% or even 99.5%?

With 99.99% you can have 4 minutes of downtime a month. If failover to a hot standby takes a minute then that shouldn't be a problem to achieve a 99.99% uptime SLO.

Re: Choose Postgres queue technology

#353

Post by sarchertech »

nsonha wrote: post #341
sarchertech wrote: post #265

Earlier quoted context omitted.

There are many times when the start order doesn’t really matter, and the additional sorting overhead isn’t worth it. In those cases people will still tend to refer to the entity holding the jobs to be processed as a queue despite the fact that it doesn’t strictly follow FIFO order. If they are being technically precise, queue isn’t the correct term, but language changes with context and time. Either way the implement…

you're confusing between "i don't care about order", and "there is no order". Name ONE queue implementation that doesn't have order.

Here you go. One of the first tutorials explaining how SKIP LOCKED works in Postgres implants a job “queue” that doesn’t have an order by clause. https://www.pgcasts.com/episodes/the-skip-locked-feature-in-...

I’m not confusing anything. I’ve seen random selection “job queues” implemented many times. As long as you truly don’t care about start order, it’s fine to trade it for increased throughout.

Re: Choose Postgres queue technology

#354

Post by worthless-trash »

worthless-trash wrote: post #141
nsonha wrote: post #122

Earlier quoted context omitted.

job should be attempted inthe same order/priority they are enqueued, that's the meaning of the word "queue". That they take varrying amounts of time is another matter.

Queue can clearly mean "work that needs to be completed" not necessarily 'work completed in order'. Your definition is much stricter than it needs to be for most use cases.

From the dictionary.

Queue: a list of data items, commands, etc., stored so as to be retrievable in a definite order, usually the order of insertion.

note the term "Usually", not "always".

Re: Choose Postgres queue technology

#355

Post by pjmlp »

latchkey wrote: post #338
pjmlp wrote: post #334

Earlier quoted context omitted.

Playing pedantic?

Ok, if you can't see what is so great about it, then I'd suggest spending some time in the GCP documentation to figure it out.

Likewise there are plenty of JMS implementations to play around.

Re: Choose Postgres queue technology

#356

Post by PH95VuimJjqBqy »

ranting-moth wrote: post #244
ritzaco wrote: post #225

I often see the "engineers copy FAANG infrastructure because they want to be cool, even though their needs are completely different" take as a kind of attack on engineers. But I think a lot of it is also about knowledge and documentation. If I want to copy FAANG or another startup, and set up an infinitely scalable queue-based architecture, I can find dozens of high quality guides, tutorials, white papers etc, showin…

> Yes maintenance is higher This is what kills you if you're a small startup. Of course it gives you a lot too. But if you're belly up then it doesn't matter. Of course go for whatever solution gives you the most benefits while not distracting you too much from your main goal. I've seen a startup where devs spent around 80% of the time fighting their tools and infrastructure. They had a 3 month runway and today there…

This line stood out to me too as very dismissive of something that can absolutely bring you to a standstill.

It's not a small thing and it's not something you should be dismissing out of hand.

Re: Choose Postgres queue technology

#357

Post by HeavyStorm »

Terrible idea. Using a database for queues means using a file format that's not optimized for this, so, unless you don't delete the rows (effectively having a always-growing table) you'll run into performance issues. Also the index for a processed flag will cause contention.

I'm certain you can work around those issues, but why if you can use a proper queue?

Re: Choose Postgres queue technology

#358

Post by latchkey »

pjmlp wrote: post #355
latchkey wrote: post #338

Earlier quoted context omitted.

Ok, if you can't see what is so great about it, then I'd suggest spending some time in the GCP documentation to figure it out.

Likewise there are plenty of JMS implementations to play around.

Sure. Name one implementation that does exactly what Cloud Tasks does.

Re: Choose Postgres queue technology

#359

Post by aaroninsf »

ekidd wrote: post #268

I've built three distributed job systems at this point. A handy rule of thumb which I have promoted for years is "build for 10x your current scale." If you need to handle 70 requests/second, design for 700. If you need to handle 20 servers running batch jobs, design for 200 servers. If you're in a startup that grows 100% per year, you'll be at 8x scale in 3 years. So you'll have time to rewrite as you grow! Out of th…

"faking transactional semantics on top of an eventually consistent data store becomes an engineering nightmare"

PREACH

If this is news to you, you just justified a week's worth of browsing HN on the clock reading this statement. Scribble it on a scrap of paper and keep it taped to your ATM card.

Re: Choose Postgres queue technology

#360

Post by xpe »

mlyle wrote: post #344
xpe wrote: post #318

Earlier quoted context omitted.

No, I won't believe it until I see evidence. Did you find it in a credible source? Which one(s)? I've found nothing credible in Merriam Webster, Etymology Online, nor _any_ other I've searched. There is at least one low-quality ad-serving site that credits ChatGPT with a definition. I'm happy to be enlightened.

e.g. Fallow's 1898 dictionary of synonyms and antonyms: https://imgur.com/a/65yLkw7 https://archive.org/download/completedictiona00falluoft There's many uses in British literature of the 1800's, and a whole lot of uses in academic literature of the 70's to 80's. https://i.imgur.com/BhMv2nF.png "Disabuse" would fit into many of these slots, but not all. Only common use now is RPG jargon; imbuing something with an attr…

Wow. Thanks!
Post reply on HN