I would love to see now PostgreSQL going more into improving developer's expierience. Job queues, job scheduler, packages.
Turning PostgreSQL into a queue serving 10k jobs per second (2013)
91–100 of 146 posts
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#92Earlier quoted context omitted.
An array can be appropriate for queuing for some things. You're wrong - as you are trying to state "a problem" has a specific solution, yet you don't know the problem. Yes, there are situations when using a dedicated message broker would be objectively better; there are many when that's totally wrong. Nothing is likely be correct about your arguments if you don't know what you are trying to fix or the sitation, which…
No, I'm assuming you haven't otherwise you'd know that redis AND postgres are inferior message brokers, both of them, and the fact that you haven't corrected me means you're not well versed at all in this problem domain.
LOL; what problem domain? We've not talked about any specific situation - which is why you're wrong; you assume far too much. I was attempting to enlighten you a little, but after a few redeliveries, it seems it's gone to DLQ.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#93Earlier quoted context omitted.
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…
I believe you are missing the point a bit here. Postgres is not the optimal tool for queueing. No one is arguing that position. However, it's also true that for a very large number of use cases, it is entirely sufficient. If you already have tooling and knowledge for managing Postgres, and Postgres is sufficient for your queueing needs, then why would you incur the cost of introducing a new service? Most situations d…
This ^ x 100.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#94[dead]
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#95The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat. Not sure if the benchmarks have been run recently or not but I’m definitely curious how t…
I actually ran them in July 2019 against QueueClassic using the Que benchmark;
queue_classic jobs per second: avg = 1879.1, max = 2072.1, min = 1779.4, stddev = 128.3
que jobs per second: avg = 1500.5, max = 1550.8, min = 1405.2, stddev = 58.1
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#96Earlier quoted context omitted.
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…
I believe you are missing the point a bit here. Postgres is not the optimal tool for queueing. No one is arguing that position. However, it's also true that for a very large number of use cases, it is entirely sufficient. If you already have tooling and knowledge for managing Postgres, and Postgres is sufficient for your queueing needs, then why would you incur the cost of introducing a new service? Most situations d…
I'm not missing the point, I'm discounting the point, because the point that adding technology is additional complexity, and complexity is bad is a nonsense argument that actually stems from a fear of new things.
This has nothing to do with getting shit done, and everything to do with people who don't want to learn new things because they're afraid of not being smart enough to make the new thing work.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#97Earlier quoted context omitted.
No, I'm assuming you haven't otherwise you'd know that redis AND postgres are inferior message brokers, both of them, and the fact that you haven't corrected me means you're not well versed at all in this problem domain.
LOL; what problem domain? We've not talked about any specific situation - which is why you're wrong; you assume far too much. I was attempting to enlighten you a little, but after a few redeliveries, it seems it's gone to DLQ.
The fact that you can't even recognize this as a problem domain means you do have zero clue what you're talking about. How about THAT?!?
Thanks for making my point for me.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#98Earlier quoted context omitted.
> These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. It's still another moving part. Thing should be as simple as they can be, but no simpler. > trying to use a database as a poor man's queue. Of course, it's not really a "poor man's" queue-- it's got some superior capabilities. It just loses on top-end performance. (Of course, using those capabilit…
It is very much a bad technology sourcing strategy, and you will kill your business if you attempt it. Period. It comes from a place of ignorance, and you're promoting ignorance. Learn why technologies exist and make an informed decision about tradeoffs, instead of being lazy and incompetent by blindly choosing technology based on what's a very locally maxima for you personally. It's selfish and damaging. Edit since…
Wow, you're super jaded on this, do you sell queuing software for a living? Also wrong: how will this kill a business, at all?
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#99Earlier quoted context omitted.
Wish I could use that in a Django app. Doesn't seem to be a viable python queueing library that allows using postgresql
I wrote django-postgres-queue for this purpose. It uses postgres transactions to keep queue and application state in sync. It also uses SKIP LOCKED to avoid some of the typical issues with using a database as a queue. https://github.com/gavinwahl/django-postgres-queue
Thanks for making this, I've been using it to help me send emails, push notifications etc. to 20-30k users in a Django project :)
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#100How does this compare to Redis? Seems like Redis would handily beat it.
The author addresses Redis: > So, many developers have started going straight to Redis-backed queues (Resque, Sidekiq) or dedicated queues (beanstalkd, ZeroMQ...), but I see these as suboptimal solutions - they're each another moving part that can fail, and the jobs that you queue with them aren't protected by the same transactions and atomic backups that are keeping your precious relational data safe and consistent.…
> Redis is rock-solid
I've lost gigabytes of data with Redis; nothing with Postgres. I run both in production.
Redis has a lot of obscure failure modes, lacks transactions (with rollback), and has no query language. For simple stuff where it's okay to lose data regularly, it's fantastic (that's when we use it).
That said, we've been doing more and more with Postgres over time, and less and less with Redis. The benefits of having all of your data in the same storage, transactionally consistent, with a query planner for ad hoc visualizations and reporting is just too great.