Apache Kafka 0.9 is released

#1

Post by nehanarkhede » Tue, Nov 24, 2015, 4:25 PM UTC

Apache Kafka 0.9 is released
confluent.io

Re: Apache Kafka 0.9 is released

#2

Post by jfim » Tue, Nov 24, 2015, 5:02 PM UTC

Congrats to the Kafka team!

The biggest changes I can see in this release are SSL support, new consumer API (beta), quotas and Kafka Connect.

Re: Apache Kafka 0.9 is released

#3

Post by simonw » Tue, Nov 24, 2015, 5:08 PM UTC

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

Re: Apache Kafka 0.9 is released

#4

Post by erichmond » Tue, Nov 24, 2015, 5:08 PM UTC

I'm always fascinated by the lack of discussion around distributed systems tooling on HN. Anyway! Congrats!

Re: Apache Kafka 0.9 is released

#5

Post by felipesabino » Tue, Nov 24, 2015, 5:11 PM UTC

I've been using kafka 0.8.2 for some time now together with Node.js for both consumer and producer.

Although the producer side is quite simple to use and have more than one option available, the consumer side there is only one project that is "maintained" and works [1][2], all other opstions either only have producer available [3] or have not received a commit in years [4].

I am a bit disappointed about how little attention Node.js with kafka had so far as there are a lot of issues on keeping connection alive and rebalancing that made it really hard to trust the system and automate zero downtime deploys.

Although I still hope all these changes in 0.9 new consumer API solve these issues, I am really happy about the decision to be backwards compatible, making the transition/upgrade a much more smooth process

> To ensure a smooth upgrade paths for our users, the 0.8 producer and consumer clients will continue to work on an 0.9 Kafka cluster.

[1] https://cwiki.apache.org/confluence/display/KAFKA/Clients#Cl...

[2] https://github.com/SOHU-Co/kafka-node/

[3] https://github.com/sutoiku/node-kafka

[4] https://github.com/wurstmeister/node-kafka-0.8-plus

Re: Apache Kafka 0.9 is released

#6

Post by felipesabino » Tue, Nov 24, 2015, 5:12 PM UTC

simonw wrote: post #3

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

Same here with node.js.

All options are too painful, either use the buggy packages available OR mix the stack with java just for the kafka bit. :(

Re: Apache Kafka 0.9 is released

#7

Post by mathnode » Tue, Nov 24, 2015, 5:13 PM UTC

erichmond wrote: post #4

I'm always fascinated by the lack of discussion around distributed systems tooling on HN. Anyway! Congrats!

The only other beast of similar nature that appears occasionally I can think of, is Onyx. Which seems pretty cool.

Anyway,Kafka Connect will provide what probably most people are looking for in Samza.

Re: Apache Kafka 0.9 is released

#8

Post by nehanarkhede » Tue, Nov 24, 2015, 5:19 PM UTC

jfim wrote: post #2

Congrats to the Kafka team! The biggest changes I can see in this release are SSL support, new consumer API (beta), quotas and Kafka Connect.

Thanks!

Re: Apache Kafka 0.9 is released

#9

Post by nehanarkhede » Tue, Nov 24, 2015, 5:23 PM UTC

felipesabino wrote: post #5

I've been using kafka 0.8.2 for some time now together with Node.js for both consumer and producer. Although the producer side is quite simple to use and have more than one option available, the consumer side there is only one project that is "maintained" and works [1][2], all other opstions either only have producer available [3] or have not received a commit in years [4]. I am a bit disappointed about how little at…

Your critique is well received. The Apache Kafka project has support for the Java clients and the non-java clients will be developed and available in a federated manner. At Confluent, we are focused on providing first class non-java clients that are API and functionality compatible with the java clients. Forthcoming releases of the Confluent Platform will include a C/C++, python and node.js client. Stay tuned http://www.confluent.io/developer#download

Re: Apache Kafka 0.9 is released

#10

Post by czinck » Tue, Nov 24, 2015, 5:25 PM UTC

simonw wrote: post #3

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

I'd say the Python library I used was borderline unusable, we stopped using Kafka (it was just a trial period, wasn't rolled to production yet) because of limits in one of the most popular Python interfaces. The interface worked well enough, the API was good, but they didn't (and the bug tracker seemed to imply they wouldn't) support synchronizing reads across processes for the same group. What's the point in a distributed synchronized log if you can't do synchronized distributed reads of the log?