11–15 of 15 posts
considering how simple sqs is, i’m confused how building your own queue system on postgres is keeping it simple. if you’re not using aws, then i understand avoiding setting up an account and some IAM bits, i guess.
If you use some other system, you still end up needing to track all of tasks and their progression in the database then identify all the ways they may have gone off track and update their state..
I'm not well-versed in the use of task queues; doesn't this make one's database layer even more of a bottleneck? Like, I thought part of the logic of a separate task queue is: 1) TQ operations, such as writing to queue, updating status, etc, involve a lot of writes; 2) If your db is your TQ, that's a lot of extra writes to the DB, which still has to handle requests from the main application code; 3) this will lead yo…
Presumably, one easily could deploy a second PG to scale Task Queues independently of the DB layer as needed. So in that respect, there is no real issue, and it might be easier to start off with a single instance and split it up as needed for scale.
The real difference is if you find yourself dealing with subpar decisions from clueless management. Not giving them (or letting them know about) the option to put the queue and DB on the same instance can save your platform from total outages (especially if the queue load is much more variable than the DB load).
What’s the max throughout anyone’s got using Postgres as a queue? And on what spec machine? Could I conceivably put 200 messages per second through a Postgres queue given a big enough RDS instance? Wondering if we don’t really need to move to Kafka at work…
Postgres can be tuned to 10k inserts per second, Kafka is definitely overkill for 200/s
What’s the max throughout anyone’s got using Postgres as a queue? And on what spec machine? Could I conceivably put 200 messages per second through a Postgres queue given a big enough RDS instance? Wondering if we don’t really need to move to Kafka at work…
I use Postgres as a queue for some bits at work using Graphile-worker [0] and it works perfectly. No need for another moving part when the data I need is in the db. Also avoids having to do outbox stuff. It’s quite cool just how far you can get with postgres [0] https://github.com/graphile/worker
I think graphile-worker as temporal.io 's little sibling.