51–60 of 144 posts
Having never actually used this platform before, does anybody know why they named it Kafka, with all the horrible meanings? Per Wiktionary, Kafkaesque: [1] 1. "Marked by a senseless, disorienting, often menacing complexity." 2. "Marked by surreal distortion and often a sense of looming danger." 3. "In the manner of something written by Franz Kafka." (like the software language was written by Franz Kafka) Example: Met…
What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Earlier quoted context omitted.
Why is that an anti-pattern? Databases have added `SKIP LOCKED` and `SELECT FOR UPDATE` to handle these use cases. What are the downsides?
as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…
(or you can just use SQS or google cloud tasks, which work out of the box)
Earlier quoted context omitted.
Why is that an anti-pattern? Databases have added `SKIP LOCKED` and `SELECT FOR UPDATE` to handle these use cases. What are the downsides?
as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…
Redesigned to create batches on the fly and then `SELECT FOR UPDATE batch SKIP LOCKED LIMIT 1` instead of `SELECT FOR UPDATE object SKIP LOCKED LIMIT 1000`. And just like that, 1000x reduction in load. Postgres is awesome.
----
The application is for processing updates to objects. Using a dedicated task queue for this is guaranteed to be worse. The objects are picked straight from their tables, based on the values of a few columns. Using a task queue would require reading these tables anyway, but then writing them out to the queue, and then invalidating / dropping the queue should any of the objects' properties update. FOR UPDATE SKIP LOCKED allows simply reading from the table ... and that's it.
Earlier quoted context omitted.
as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…
40,000 per second is waaaaay beyond where you should use a dedicated queuing solution. Even dedicated queues require tuning to handle that kind of throughput. (or you can just use SQS or google cloud tasks, which work out of the box)
Earlier quoted context omitted.
as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…
I believe the article and parent comment were discussing queue solutions for low-volume situations.
Earlier quoted context omitted.
as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…
> 150 million objects / hour Is not a low volume unless this could be done in batches of hundreds.
Earlier quoted context omitted.
40,000 per second is waaaaay beyond where you should use a dedicated queuing solution. Even dedicated queues require tuning to handle that kind of throughput. (or you can just use SQS or google cloud tasks, which work out of the box)
I hit 60k per second in 2020 on a 2-core, 100GB SSD installation of PG on GCP. And "tuning" PG is way easier than any dedicated queueing system I've seen. Does there exist a dedicated queueing system with an equivalent to EXPLAIN (ANALYZE)?
https://softwaremill.com/mqperf/
Maybe you should write a letter?
Kafka for small message volumes is one of those distinct resume-padding architectural vibes.
Earlier quoted context omitted.
as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…
I did sth similar. Designed and built for 10 million objects / hour. Picked up by workers in batches of 1k. Benchmark peaked above 200 million objects / hour with PG in a small VM. Fast forward two years, the curse of success strikes, and we have a much higher load than designed for. Redesigned to create batches on the fly and then `SELECT FOR UPDATE batch SKIP LOCKED LIMIT 1` instead of `SELECT FOR UPDATE object SKI…