Show HN: KQ – Simple Job Queue for Python Using Kafka
1–10 of 31 posts
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#2Getting a Permission Denied page for the docs at http://kq.readthedocs.io/en/master/
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#3> It is backed by Apache Kafka and designed primarily for ease of use.
That's a bold claim.
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#4One of the biggest problems with treating Kafka as a job queue is that you suffer from head-of-line blocking. Kafka doesn't expose per-message visibility/acknowledgement semantics like RabbitMQ/Redis PUSH+POP/SQS does. Each consumer group tracks offsets into the partitions of a log (aka a topic). This offset is just a number that points to a specific message in the Kafka partition. If you get stuck on message 123, you either can't proceed to 124, proceed and don't commit your offset but risk replaying 124, or skip 123.
A great many of our services publish to Kafka, those consuming services which seek to treat individual records as tasks (or bundles of tasks) as opposed to a linear log must either skip failures or push them onto SQS for background retry. Our batching consumers have to track out-of-order completion of work and commit up to the lowest completed offset, meaning a slow task can delay offset commits. If a consumer is stopped before finishing that slow task, we have to replay work which means all work has to be idempotent. In practice, it works well enough, but it's still some gymnastics.
I suspect this is why Google invested so much into making PubSub scalable despite per-message semantics. It's considerably simpler in many ways, even if you have to bake in your own ordering/monotonicly increasing identifiers.
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#5One of the biggest problems with treating Kafka as a job queue is that you suffer from head-of-line blocking. Kafka doesn't expose per-message visibility/acknowledgement semantics like RabbitMQ/Redis PUSH+POP/SQS does. Each consumer group tracks offsets into the partitions of a log (aka a topic). This offset is just a number that points to a specific message in the Kafka partition. If you get stuck on message 123, yo…
Very true. I indeed found the lack of visibility into per-message information very painful when I was building this. One way I tried to alleviate the issue was providing a consumer "callback" to make it easier for users to plug their own code in to handle job failures (like your example of using SQS).
I've also thought about reserving a topic + consumer group specifically for failed jobs and bake the retry logic into KQ itself. But that's an area I must explore more.
I'm not sure if I understand what you are saying about batching consumers. What do you mean by batching in this context? Thanks for your input.
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#6Getting a Permission Denied page for the docs at http://kq.readthedocs.io/en/master/
Fixed!
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#7> It is backed by Apache Kafka and designed primarily for ease of use. That's a bold claim.
He means the software requires Kafka as a dependency, not that the foundation is sponsoring it. The claim of "designed primarily for ease of use" is a personal one, not hard to make.
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#8Anyone have any opinions on the long-term viability of Kafka? I've been lurking on the kafka dev mailing lists and I'm fairly turned off by the attitude from Confluent's employees that I've read from. There was a recent thread about their open source status and how they backhand Apache's open source philosophies, I'm wondering if they are thinking of moving away from open source in the future.
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#9> It is backed by Apache Kafka and designed primarily for ease of use. That's a bold claim.
He means the software requires Kafka as a dependency, not that the foundation is sponsoring it. The claim of "designed primarily for ease of use" is a personal one, not hard to make.
I read the comment as "simple and ease of use are not terms often associated with Kafka".
Re: Show HN: KQ – Simple Job Queue for Python Using Kafka
#10Pretty clean code. Nice job.