I always thought that using a database server for RPC was considered an anti-pattern.
Turning PostgreSQL into a queue serving 10k jobs per second (2013)
41–50 of 146 posts
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#42The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat. Not sure if the benchmarks have been run recently or not but I’m definitely curious how t…
Wish I could use that in a Django app. Doesn't seem to be a viable python queueing library that allows using postgresql
[deleted]
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#43The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat. Not sure if the benchmarks have been run recently or not but I’m definitely curious how t…
Wish I could use that in a Django app. Doesn't seem to be a viable python queueing library that allows using postgresql
rq is pretty good, have you checked it out?
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#44I have also found the lack of transactional guarantees in typical job queues to be very problematic. One problem with using PostgreSQL in this way (using either advisory locks or LOCK FOR UPDATE) is that it requires you to keep an open connection to the database whilst the job is being worked on. For a MySQL database, this would be just fine, but PostgreSQL uses a process-per-connection model which caps the number of…
I think the biggest bonus for me of transactional queues like this is knowing that either everything worked - the inserted/updated/deleted things, and enqueuing of any jobs - or nothing did. Whilst using resque, or anything else outside of postgres, occasionally things would rollback but have sent jobs. This sucks; as you either get errors or unexpected things, or have to code around it to fix that. FYI, we've been u…
Did you guys ever evaluate Que? What made you choose QueueClassic?
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#45How does this compare to Redis? Seems like Redis would handily beat it.
You may be right in terms of performance (I've no idea) but article states: > many developers have started going straight to Redis-backed queues (Resque, Sidekiq) or dedicated queues (beanstalkd, ZeroMQ...), but I see these as suboptimal solutions - they're each another moving part that can fail, and the jobs that you queue with them aren't protected by the same transactions and atomic backups that are keeping your p…
I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job."
Edit: I'm rate limited so to elaborate a bit.
Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a specific problem.
Using a RDBMS for queueing is not what it was built to do, and you will run into issues doing so (I have).
By trying to use one tool for everything, you lose out on all kinds of optimizations, features, and performance enhancements that are specific to the problem you're trying to solve.
It's bad engineering to try and force Postgres into the role of queue when vastly superior technologies exist. You're actively hurting the engineers you work with, and the company you work for, if you force the same tech into use cases it isn't optimal for.
I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#46About 22 years back I got in trouble for recommending we drop Tuxedo Queue with just a database table since we can get all the transactional options for free, just the same, without all the headaches an operational issues involved in connecting all our code with the C++ middleware, especially since Java and PHP were in the mix and Tuxedo was kinda of in it's own world and was pretty much the only non-free software we were using and using in a horrible idiotic way.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#47Bad idea to use that because if the worker crash the event is lost, don't use PG for that.
This is pubsub and the data is in an ACID compliant database. The worker will just re-subscribe and poll for events on startup.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#48Now compare to OpenAMQ, ZeroMQ, RabbitMQ, NSQ, Kafka. I have seen benchmarks reaching millions of messages per second.
The article is discussing job queues, not messaging. The MQ systems plus Kafka that you are referring to are message transport systems.
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#49Bad idea to use that because if the worker crash the event is lost, don't use PG for that.
> Bad idea to use that because if the worker crash the event is lost
The only case where this would be problematic is if the worker crashed after doing some side effect external to the queue server that rendered the job non-idempotent, but before committing an acknowledgement. But that's not really a “what you use as a queue server” issue as a “are you also using a distributed transaction system to coordinate all side effects including queue updates” issue. (If the same Postres DB is the operational DB and the queue server, you may be able to avoid distributed transactions by having all the side effects in the DB, though that creates its own issues.)
Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)
#50Earlier quoted context omitted.
I think the biggest bonus for me of transactional queues like this is knowing that either everything worked - the inserted/updated/deleted things, and enqueuing of any jobs - or nothing did. Whilst using resque, or anything else outside of postgres, occasionally things would rollback but have sent jobs. This sucks; as you either get errors or unexpected things, or have to code around it to fix that. FYI, we've been u…
Did you guys ever evaluate Que? What made you choose QueueClassic?
We did; but I think it came around after we started. It was faster at that time, we looked at also it's well written - we'd probably have used it if it was out, but we didn't really mind about the speed - we're not latency sensitive for what we're doing. Fast forward a little, and we now maintain QueueClassic, and it's now comparable / slightly faster.