21–30 of 31 posts
Why should I use this over the simplier RQ?
Earlier quoted context omitted.
Or the proven Celery
This Celery? https://news.ycombinator.com/item?id=12844206
For that specific bug in question, does KQ have logic around DST switch? I was not able to find any. So it looks like you would have the same bug in KQ, right?
Earlier quoted context omitted.
I am not associated with the Kafka dev team or Confluent in any way but have been using Kafka since their 0.7.x days. Speaking of their open source commitment so far, I haven't seen any kind of problems in the way they deal with the community. They have been open to bug fixes, feature enhancements and other contributions. Many of the Confluent employees you see currently, started off by contributing to the open sourc…
Yes, I think that's the one but my biggest takeaway from what I remember was "Apache is okay but..." I remember reading about half a dozen Confluent employees towing the party line and parroting the exact same argument, about how being under Apache was stifling their innovation. And yet Kafka is thriving under Apache, so their arguments smelled fishy as hell. It sounds like they are setting up the foundation to pull…
What kind of jobs do people put into such queues? Because I've only seen job queues used as a poor man's RPC brokers and (poor?) replacement for crontab. I think I may miss some use cases where this is a valid choice.
What kind of jobs do people put into such queues? Because I've only seen job queues used as a poor man's RPC brokers and (poor?) replacement for crontab. I think I may miss some use cases where this is a valid choice.
If you don't like it don't use it. I read through your profile's comments, all you do is complain and bash other peoples ideas.
For our web application, we're already in the middle of transition from Celery to a proper RPC system, so your comment (aimed to, I don't know, make me ashamed? make me shut up and go doing some real work?) somewhat missed.
Though I really want to know if there are any sensible uses for task queue, specifically with web applications in mind, as most people seem to use task queues for that.
> [...] all you do is complain and bash other peoples ideas.
Only the stupid or incomplete ones, like somebody boasting about their system without describing what does it do or how to install and use it.
Anyone have any opinions on the long-term viability of Kafka? I've been lurking on the kafka dev mailing lists and I'm fairly turned off by the attitude from Confluent's employees that I've read from. There was a recent thread about their open source status and how they backhand Apache's open source philosophies, I'm wondering if they are thinking of moving away from open source in the future.
That was a super silly discussion and I'd be annoyed reading it too. It does not, however, represent any kind of "move away from open source" by Confluent.
The discussion was not actually about open source vs non-open source but rather around how to curate the Kafka ecosystem. Currently there are several hundred independent open source projects that act as connectors, clients, monitoring tools, processing libraries and frameworks, etc. Some of these are fantastic and active communities, some are built by companies that use Kafka, some by vendors that use Kafka in a product in some way. This has always been the structure of this ecosystem, since even before we moved Kafka into Apache. These projects are mostly Apache licensed and on github, but mostly not part of the ASF. You can see a subsample of them here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
The proposal from a group at Hortonworks was to duplicate one of the modules Confluent built, a REST layer, and build a new one in the main project. I don't think this is inherently a bad thing to do, but there wasn't really any concrete rationale...there was no complaint with the code in that module, nor any kind of complaint with the governance, nor any complaint with the community, etc. As a result a lot of us at Confluent kind of felt it was an odd exercise that had more to do with Hortonworks' policy of only shipping Apache projects, which in turn comes out of the weird fractious politics of the Hadoop ecosystem. The reality is that policy just doesn't fit with the Kafka ecosystem as it stands today.
One of the biggest problems with treating Kafka as a job queue is that you suffer from head-of-line blocking. Kafka doesn't expose per-message visibility/acknowledgement semantics like RabbitMQ/Redis PUSH+POP/SQS does. Each consumer group tracks offsets into the partitions of a log (aka a topic). This offset is just a number that points to a specific message in the Kafka partition. If you get stuck on message 123, yo…
Very true. I indeed found the lack of visibility into per-message information very painful when I was building this. One way I tried to alleviate the issue was providing a consumer "callback" to make it easier for users to plug their own code in to handle job failures (like your example of using SQS). I've also thought about reserving a topic + consumer group specifically for failed jobs and bake the retry logic into…
Our batching consumers provide pluggable behavior for handling a failing batch, but usually it's pushed onto SQS since those can cycle around a few times until we notice and fix whatever condition is preventing progress on that work.
* - 123 actually, as if you commit offset 123 the consumer will fetch offset 123 again on start, but that's implementation esoterica
Earlier quoted context omitted.
This Celery? https://news.ycombinator.com/item?id=12844206
Dunno why people would blame the Celery devs for that bug - if you're setting the clock on production servers to a non-monotonic TZ you are going to have a bad time.
Also, I would understand if it were tasks between 2AM and 3AM that run twice in transition from summer time to winter time (there are two 2:30 that day, after all), but the bug affected tasks scheduled for the middle of the day.