141–150 of 209 posts
It's really simple. I used this snippet for a real-time application which can react to any kind of change in the database instantly.
Earlier quoted context omitted.
I assume that is their main database for everything, not just for pub/sub. One of the big benefits of doing it that way is that you have proper transaction handling across jobs and their related data.
Come on man… you can run the whole thing off if a few Gb instance. Such a huge instance should be able to do about 100k a second!
Earlier quoted context omitted.
Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!
But Kafka does significantly more. And if your needs are simpler like in this case then there are dozens of smaller pub/sub/queue systems that you could compare this to.
Earlier quoted context omitted.
This assumes that you're creating a transaction per message, which I think is not advisable.
if you care to elaborate, i'm curious -- what alternative(s) would you recommend instead of one transaction per message, and why?
Earlier quoted context omitted.
If a job fails, the connection to the database will timeout. Postgres will rollback the transaction, which releases row locks, freeing a job to be retried. Of course, the database client and server together form a distributed system. The client might continue processing a job under the mistaken impression that it still holds the lock. Jobs still need to be idempotent, as with the streaming platforms.
This assumes that you're creating a transaction per message, which I think is not advisable.
Anybody using graphile-worker[1] in production/heavy load? It looks awesome, and I coded up some simple prototype tasks (email, sms, etc), but question how it truly scales. They claim horizontal scaling is trivial. > graphile-worker is horizontally scalable. Each instance has a customisable worker pool, this pool defaults to size 1 (only one job at a time on this worker) but depending on the nature of your tasks (i.e…
Regarding the horizontal scalability; that relates to if you have heavy tasks (tasks that take a second or more to execute) - you can use more instances to get higher throughput.
Hope this helps!
Earlier quoted context omitted.
> doing hundreds of thousands of messages per day > The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. Am I the only one?
Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!
Earlier quoted context omitted.
Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!
It's that much on a popular cloud platform, you can buy this for 3-4 times that amount and use it for years.
Earlier quoted context omitted.
If a job fails, the connection to the database will timeout. Postgres will rollback the transaction, which releases row locks, freeing a job to be retried. Of course, the database client and server together form a distributed system. The client might continue processing a job under the mistaken impression that it still holds the lock. Jobs still need to be idempotent, as with the streaming platforms.
This assumes that you're creating a transaction per message, which I think is not advisable.
Transactions in MVCC are relatively cheap. The main resource of contention is a global txid that can disastrously wrap around if autovacuum is disabled. That process is responsible for a few other important tasks, like updating statistics for the query planner and maintaining BRIN indexes.