Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
1–10 of 15 posts
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#2I use Postgres as a queue for some bits at work using Graphile-worker [0] and it works perfectly. No need for another moving part when the data I need is in the db. Also avoids having to do outbox stuff.
It’s quite cool just how far you can get with postgres
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#3What’s the max throughout anyone’s got using Postgres as a queue? And on what spec machine?
Could I conceivably put 200 messages per second through a Postgres queue given a big enough RDS instance?
Wondering if we don’t really need to move to Kafka at work…
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#4I'm not well-versed in the use of task queues; doesn't this make one's database layer even more of a bottleneck?
Like, I thought part of the logic of a separate task queue is: 1) TQ operations, such as writing to queue, updating status, etc, involve a lot of writes; 2) If your db is your TQ, that's a lot of extra writes to the DB, which still has to handle requests from the main application code; 3) this will lead you to have to scale your DB layer in some fashion earlier than you would otherwise have to, and scaling the DB layer via, let's say sharding or something, is harder/trickier than setting up or scaling a task queue.
Especially if early/small, who wants to worry about scaling their DB layer unless they have to?
Would appreciate some insight; I have a side project that involves downloading and processing a lot of PDFs; if it sees the light of day, it will need a task queue of some kind. I'm already using Postgres, so it would be tempting to use that as the TQ, so long as that doesn't require more careful management to get working right.
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#5I'm not well-versed in the use of task queues; doesn't this make one's database layer even more of a bottleneck? Like, I thought part of the logic of a separate task queue is: 1) TQ operations, such as writing to queue, updating status, etc, involve a lot of writes; 2) If your db is your TQ, that's a lot of extra writes to the DB, which still has to handle requests from the main application code; 3) this will lead yo…
> Especially if early/small, who wants to worry about scaling their DB layer unless they have to?
If early/small, who wants to worry about putting another piece of infrastructure (RabbitMQ, Redis, etc) in your stack that you have to operate?
It will probably surprise you how far you can go with just Postgres.
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#6What’s the max throughout anyone’s got using Postgres as a queue? And on what spec machine? Could I conceivably put 200 messages per second through a Postgres queue given a big enough RDS instance? Wondering if we don’t really need to move to Kafka at work…
Postgres can be tuned to 10k inserts per second, Kafka is definitely overkill for 200/s
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#7I'm not well-versed in the use of task queues; doesn't this make one's database layer even more of a bottleneck? Like, I thought part of the logic of a separate task queue is: 1) TQ operations, such as writing to queue, updating status, etc, involve a lot of writes; 2) If your db is your TQ, that's a lot of extra writes to the DB, which still has to handle requests from the main application code; 3) this will lead yo…
> Especially if early/small, who wants to worry about scaling their DB layer unless they have to? If early/small, who wants to worry about putting another piece of infrastructure (RabbitMQ, Redis, etc) in your stack that you have to operate? It will probably surprise you how far you can go with just Postgres.
> If early/small, who wants to worry about putting another piece of infrastructure (RabbitMQ, Redis, etc) in your stack that you have to operate?
Yeah, I get that. I guess my assumption is kind of that these things are so common and mature at this point, that setting them up isn't that big of a deal. I suppose too that if one is early/small, one should probably use a managed solution for anything possible as long as it isn't crazy expensive; so a managed TQ should be easy to set up, and if you're using managed Postgres, scaling it horizontally should also be relatively easy (?)
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#8considering how simple sqs is, i’m confused how building your own queue system on postgres is keeping it simple.
if you’re not using aws, then i understand avoiding setting up an account and some IAM bits, i guess.
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#9considering how simple sqs is, i’m confused how building your own queue system on postgres is keeping it simple. if you’re not using aws, then i understand avoiding setting up an account and some IAM bits, i guess.
If you already have Postgres as a dependency, then adding a second is almost always more complicated than just using your existing one.
Re: Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?
#10considering how simple sqs is, i’m confused how building your own queue system on postgres is keeping it simple. if you’re not using aws, then i understand avoiding setting up an account and some IAM bits, i guess.
If you use some other system, you still end up needing to track all of tasks and their progression in the database then identify all the ways they may have gone off track and update their state..