I'm not very knowledgable about db internals so sorry if this comes off as ignorant, but in an era where cpus execute billions of instructions per second per core, is 10000 jobs per second supposed to be impressive? Is this kind of problem bottlenecked by memory?
Turning PostgreSQL into a queue serving 10k jobs per second (2013)
81–90 of 146 posts
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#82PostgreSQL never ceases to amaze me.
The great thing is that it is a mature tool and you don't need to bring another beast to the zoo.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#83I'm not very knowledgable about db internals so sorry if this comes off as ignorant, but in an era where cpus execute billions of instructions per second per core , is 10000 jobs per second supposed to be impressive? Is this kind of problem bottlenecked by memory?
[deleted]
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#84Earlier quoted context omitted.
It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends.…
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…
I believe you are missing the point a bit here. Postgres is not the optimal tool for queueing. No one is arguing that position. However, it's also true that for a very large number of use cases, it is entirely sufficient.
If you already have tooling and knowledge for managing Postgres, and Postgres is sufficient for your queueing needs, then why would you incur the cost of introducing a new service? Most situations don't require the best tool for the job, and in that case, you are well served to use a tool you already have.
If you are using an abstraction layer over your queues (you almost certainly are) then you should be able to easily change to a different queue implementation later, should the need arise.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#85How does this compare to Redis? Seems like Redis would handily beat it.
Because Que has transactional/ACID guarantees, and Redis does not.
Actually, redis makes several guarantees. Just not as acid as rds. Source: https://redis.io/topics/transactions
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#86Earlier quoted context omitted.
It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends.…
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…
Earlier you said something about making excuses for learning and now, claim something subjective is 100% true and blatently obvious -- and that could block your own learning.
Like many things in this space there are a number of good ways - each with minor trade offs
Others posters have detailed those trades very clearly.
So now we can all be more educated about which one of the six good ways we'd choose.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#87Earlier quoted context omitted.
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…
An array can be appropriate for queuing for some things. You're wrong - as you are trying to state "a problem" has a specific solution, yet you don't know the problem. Yes, there are situations when using a dedicated message broker would be objectively better; there are many when that's totally wrong. Nothing is likely be correct about your arguments if you don't know what you are trying to fix or the sitation, which…
No, I'm assuming you haven't otherwise you'd know that redis AND postgres are inferior message brokers, both of them, and the fact that you haven't corrected me means you're not well versed at all in this problem domain.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#88The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat. Not sure if the benchmarks have been run recently or not but I’m definitely curious how t…
Wish I could use that in a Django app. Doesn't seem to be a viable python queueing library that allows using postgresql
I wrote django-postgres-queue for this purpose. It uses postgres transactions to keep queue and application state in sync. It also uses SKIP LOCKED to avoid some of the typical issues with using a database as a queue.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#89Now compare to OpenAMQ, ZeroMQ, RabbitMQ, NSQ, Kafka. I have seen benchmarks reaching millions of messages per second.
i don't need millions messages per second for my job queues, i'm not google.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#90Earlier quoted context omitted.
> These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. It's still another moving part. Thing should be as simple as they can be, but no simpler. > trying to use a database as a poor man's queue. Of course, it's not really a "poor man's" queue-- it's got some superior capabilities. It just loses on top-end performance. (Of course, using those capabilit…
It is very much a bad technology sourcing strategy, and you will kill your business if you attempt it. Period. It comes from a place of ignorance, and you're promoting ignorance. Learn why technologies exist and make an informed decision about tradeoffs, instead of being lazy and incompetent by blindly choosing technology based on what's a very locally maxima for you personally. It's selfish and damaging. Edit since…
Bruh, folks are making an informed decision - and landing on PostgreSQL. Their scenario/conclusion is just different than yours.
It's not lazy, blind or selfish as you claim - and frankly when you disagree with authority & condescended tone - it's a turn off.