Shameless plug: if you need to send application logs to Kafka, consider Flowgger: https://github.com/jedisct1/flowgger
Apache Kafka 0.9 is released
31–40 of 40 posts
Re: Apache Kafka 0.9 is released
#32I've been using kafka 0.8.2 for some time now together with Node.js for both consumer and producer. Although the producer side is quite simple to use and have more than one option available, the consumer side there is only one project that is "maintained" and works [1][2], all other opstions either only have producer available [3] or have not received a commit in years [4]. I am a bit disappointed about how little at…
Your critique is well received. The Apache Kafka project has support for the Java clients and the non-java clients will be developed and available in a federated manner. At Confluent, we are focused on providing first class non-java clients that are API and functionality compatible with the java clients. Forthcoming releases of the Confluent Platform will include a C/C++, python and node.js client. Stay tuned http://…
Is anyone working on Go bindings for 0.9?
Re: Apache Kafka 0.9 is released
#33I'm always fascinated by the lack of discussion around distributed systems tooling on HN. Anyway! Congrats!
The only other beast of similar nature that appears occasionally I can think of, is Onyx. Which seems pretty cool. Anyway,Kafka Connect will provide what probably most people are looking for in Samza.
Haha, funny you mention Onyx! We pipe data from kafka into storm :) Great combo!
Re: Apache Kafka 0.9 is released
#34The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.
Same problem for .NET/C#. Nothing established/built enough to feel comfortable using it in production.
While it feels a bit hacky and unclean, you may want to try using IKVM (http://www.ikvm.net/) to translate and import the Java client in to your .NET project.
Given the difficulty in building a client period (distributed systems, race conditions, etc), being able to rely on the widely adopted & supported official client is quite attractive.
In my test cases the performance is on par running natively on the JVM, except when compression is enabled.
Another option is using the REST proxy and accepting the trade-offs that imposes.
Re: Apache Kafka 0.9 is released
#35What is the use case for this product? Could you use it as a replicating database across sites?
"What is the use case for a horizontally scalable message broker?"
That promises durability, at least once delivery and sequential consistency (an important set of promises that put it largely in a class by itself).
Re: Apache Kafka 0.9 is released
#36Earlier quoted context omitted.
Same here with node.js. All options are too painful, either use the buggy packages available OR mix the stack with java just for the kafka bit. :(
Hence why I use Groovy for any Kafka endeavors.
The same problem exists with Python, C#, node.js, and Groovy.
Re: Apache Kafka 0.9 is released
#37Earlier quoted context omitted.
I'd say the Python library I used was borderline unusable, we stopped using Kafka (it was just a trial period, wasn't rolled to production yet) because of limits in one of the most popular Python interfaces. The interface worked well enough, the API was good, but they didn't (and the bug tracker seemed to imply they wouldn't) support synchronizing reads across processes for the same group. What's the point in a distr…
Sounds like old news, but if this is still an issue, PyKafka does allow balanced reads across a consumer group. https://github.com/parsely/pykafka
Yeah, it's no longer relevant for that project, but I like the ideas behind Kafka and will probably use it again so I'll look at PyKafka before I look at kafka-python in the future.
Re: Apache Kafka 0.9 is released
#38What is the use case for this product? Could you use it as a replicating database across sites?
Kafka can be used if you want to treat data as streams for some processing (think producer-consumer kind of scenarios). You can point to the stream from any point in time to read it 'as and when things happened'. Kafka's own nodes have replication enabled, and the data that it produces can be consumed in a distributed setting as well (meaning multiple consumers acting as a single high level consumer). But it is not a traditional database as MongoDB or MySQL.
Re: Apache Kafka 0.9 is released
#39Were the diagrams done with software, or hand drawn? If software, I'm curious what package/style you used, the style looks very similar to Martin Kleppmann's presentation at StrangeLoop; I assumed his were hand drawn but I'm realizing now this might be a omni style or something.
This always comes up with Martin Kleppman diagrams. See this discussion: https://news.ycombinator.com/item?id=9613118 The bottom comments seem to agree it was done using Paper.
From the Kleppmann's mouth:
https://twitter.com/martinkl/status/629169643710775296
I saw him talk at a conference, and that was one of the questions someone asked. He must be so fed up with it by now!
Re: Apache Kafka 0.9 is released
#40Congrats on the release!
@nehanarkhede was there a specific reason the new consumer is written in Java? The previous consumers are all written in Scala.