Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning PostgreSQL into a queue serving 10k jobs per second (2013)

gist.github.com

Post reply on HN
Print view View original

71–80 of 146 posts

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#71

Post by ukd1 »

mlyle wrote: post #54
dimino wrote: post #45

Earlier quoted context omitted.

I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job." Edit: I'm rate limited so to elaborate a bit. Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a s…

Fewer pieces in the technology stack is better, because each piece is operationally expensive (it is its own set of work in maintenance, upgrades, migrations, etc). There are also the mentioned costs (you can't atomically do things across divergent pieces of infrastructure, at least not very easily at all). Fighting the tendency of everyone wanting to draw in each new interesting bit of technology is important. Every…

So much love for this; 'Every piece of complexity you incur should be proven as required', plus 'Fewer pieces in the technology stack is better, because each piece is operationally expensive'. Also, this totally ignores (as these, imho are enough in themselves) the actual practical differences between using a single thing and two - synchronizing state; most of the cases here guarantee atomicnes individually, but not as a pair - you (the engineer) have to deal with that - which is non-trivial.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#72

Post by nicoburns »

codetrotter wrote: post #67
elamje wrote: post #55

Earlier quoted context omitted.

Recently was wishing this myself. I’ve been using rq, which although I hear is much faster than an rdbms is also another dependency and service to worry about failing. Trying to keep things simple is nice, I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains

> I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains If it is possible to set up a smaller test environment with one or a few instances of your DB and Redis, and you have time to do it, you could do some testing where you purposely shut down each of them at various points in time and inspect what happens to your app…

docker-compose is excellent for these kind of testing setups.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#73

Post by ukd1 »

codetrotter wrote: post #67
elamje wrote: post #55

Earlier quoted context omitted.

Recently was wishing this myself. I’ve been using rq, which although I hear is much faster than an rdbms is also another dependency and service to worry about failing. Trying to keep things simple is nice, I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains

> I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains If it is possible to set up a smaller test environment with one or a few instances of your DB and Redis, and you have time to do it, you could do some testing where you purposely shut down each of them at various points in time and inspect what happens to your app…

I mean, I love testing, but this is a great example of why you should just use less things if you can. Combinations of failure modes aren't fun to get right.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#74

Post by arcticfox »

ukd1 wrote: post #38

This post is pretty out of date; Que and QueueClassic moved to SKIP LOCKED at some point, over using advisory locks / lock head methods. It's much faster, but only supported in Postgres >= 9.5.

Que still doesn't use SKIP LOCKED, unfortunately. I think QueueClassic is just the better library at this point, as SKIP LOCKED simplifies everything tremendously and offers similar performance.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#75

Post by dimino »

ukd1 wrote: post #70
dimino wrote: post #45

Earlier quoted context omitted.

I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job." Edit: I'm rate limited so to elaborate a bit. Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a s…

It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends.…

Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing.

You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics.

I cannot stress this enough; you are flat wrong if you think Postgres is appropriate to use for queueing.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#76

Post by dimino »

mlyle wrote: post #66
threeseed wrote: post #65

Earlier quoted context omitted.

These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. And then you have a component that is designed for the job instead of trying to use a database as a poor man's queue.

> These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. It's still another moving part. Thing should be as simple as they can be, but no simpler. > trying to use a database as a poor man's queue. Of course, it's not really a "poor man's" queue-- it's got some superior capabilities. It just loses on top-end performance. (Of course, using those capabilit…

It is very much a bad technology sourcing strategy, and you will kill your business if you attempt it. Period.

It comes from a place of ignorance, and you're promoting ignorance. Learn why technologies exist and make an informed decision about tradeoffs, instead of being lazy and incompetent by blindly choosing technology based on what's a very locally maxima for you personally. It's selfish and damaging.

Edit since I'm rate limited: The folks landing on Postgres are not landing there after due consideration, they're landing on a technology they're familiar with because they don't know how to learn.

It is lazy, ignorant and selfish, and while of course people don't like having truth spoken to them, that's what it is; truth.

Edit 2: Making these poor technology choices will kill your business because you won't have the agility that using a specific message broker technology gives you. You won't have built-in solutions to common queueing problems, you won't have dedicated specific logging/metrics to monitor, you won't have the libraries in your preferred language to directly tackle your problem, you won't have the support community available to you (it will be much smaller), you won't be able to pivot onto related patterns as your needs change, your scaling will always be more complicated because fewer people are doing it and the tool you use isn't tailored for your use case; the list goes on. Your competitors will swallow you because they move faster than you do, and your business will die.

This is annoying to me because I've been in situations where I've had to maintain and write features against technology that was a poor fit for its use, but people like those in this comment sections bitched and moaned about a "new thing" existing in our stack. The reality was they didn't want to learn anything, and were fine pushing the hard work off onto the developers, so they could safely continue to do as little as they possibly could get away with.

Do your job, learn technology that actually fits your use case, and stop trying to push work off onto other people.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#78

Post by cwalv »

Another potential benefit to use a RDBMS system as a queue is that it can make it much simpler to express priorities in cases where it's not a simple FIFO, i.e., if the next job a consumer should take depends on more than just the time the job was added to the queue.

One place this has come up for me is when the next job that's picked depends on currently running jobs, e.g., each job is associated with a user, and if a single user already has N tasks running you may want to prioritize another user's tasks for the N+1 slot.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#79

Post by ukd1 »

dimino wrote: post #75
ukd1 wrote: post #70

Earlier quoted context omitted.

It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends.…

Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…

An array can be appropriate for queuing for some things. You're wrong - as you are trying to state "a problem" has a specific solution, yet you don't know the problem. Yes, there are situations when using a dedicated message broker would be objectively better; there are many when that's totally wrong.

Nothing is likely be correct about your arguments if you don't know what you are trying to fix or the sitation, which should also be blatantly obvious. You're applying some unstated subjective situation in your head, then dumping out something you've heard / done before. LOL.

Also "if you've ever used both" - I have, which is why I know. It's gray; sometimes you should, sometimes you shouldn't. You're assuming I haven't as I have differing point of view to you.

Post reply on HN