Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Kafka 0.9 is released

confluent.io

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Apache Kafka 0.9 is released

#11

Post by nehanarkhede »

mathnode wrote: post #7
erichmond wrote: post #4

I'm always fascinated by the lack of discussion around distributed systems tooling on HN. Anyway! Congrats!

The only other beast of similar nature that appears occasionally I can think of, is Onyx. Which seems pretty cool. Anyway,Kafka Connect will provide what probably most people are looking for in Samza.

That observation is correct. Currently, people misuse stream processing systems like Storm and Samza for data import/export. This is an overkill. Kafka Connect is focused on providing scalable and operational connectors to various systems using Kafka as the underlying transport mechanism.

Re: Apache Kafka 0.9 is released

#12

Post by hoffcoder »

I have been using Kafka 0.8.2 in a production setting for consuming real-time event traffic from our caching layer for six months. The most difficult parts of my experience were the occasional consumer lags that erupted without warning/cause in the high level Java consumer APIs. A lot of experimentation with their configuration proved to be futile and now we have had to create a feedback system that triggers alerts to change group Ids of our high level consumers every time some consumers start lagging.

Otherwise the performance of Kafka has been impressive (giving a throughput of upto 15000 packets/sec to a 8-consumer pool), even though I have not had the chance to compare it with any other such tool/library.

Nevertheless, I think this update is a long awaited one, and Kafka Connect may really be good starting point for building more (and better) endpoints.

Re: Apache Kafka 0.9 is released

#13

Post by pimeys »

nehanarkhede wrote: post #9
felipesabino wrote: post #5

I've been using kafka 0.8.2 for some time now together with Node.js for both consumer and producer. Although the producer side is quite simple to use and have more than one option available, the consumer side there is only one project that is "maintained" and works [1][2], all other opstions either only have producer available [3] or have not received a commit in years [4]. I am a bit disappointed about how little at…

Your critique is well received. The Apache Kafka project has support for the Java clients and the non-java clients will be developed and available in a federated manner. At Confluent, we are focused on providing first class non-java clients that are API and functionality compatible with the java clients. Forthcoming releases of the Confluent Platform will include a C/C++, python and node.js client. Stay tuned http://…

We've been using Kafka for almost two years already. Still lots of our codebase is Ruby, and Poseidon is not really that great client library. It's slow and it's not threadsafe.

In our Scala-side we're happy with the current offering.

Re: Apache Kafka 0.9 is released

#14

Post by ora600 »

hoffcoder wrote: post #12

I have been using Kafka 0.8.2 in a production setting for consuming real-time event traffic from our caching layer for six months. The most difficult parts of my experience were the occasional consumer lags that erupted without warning/cause in the high level Java consumer APIs. A lot of experimentation with their configuration proved to be futile and now we have had to create a feedback system that triggers alerts t…

In this case, you will enjoy the new consumer in 0.9 a lot!

Re: Apache Kafka 0.9 is released

#15

Post by joshbaptiste »

felipesabino wrote: post #6
simonw wrote: post #3

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

Same here with node.js. All options are too painful, either use the buggy packages available OR mix the stack with java just for the kafka bit. :(

Hence why I use Groovy for any Kafka endeavors.

Re: Apache Kafka 0.9 is released

#16

Post by emmett9001 »

simonw wrote: post #3

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

https://github.com/parsely/pykafka

PyKafka is currently used in production at Parse.ly, and I've gotten feedback from a lot of other folks who are using it in production as well. The big benefit over kafka-python is that PyKafka supports multi-consumer groups that balance consumption via ZooKeeper with its BalancedConsumer interface. See this thread ( https://github.com/Parsely/pykafka/issues/334 ) for more detail on the differences between the two libraries.

The PyKafka project is prioritizing support for Kafka 0.9 in the next few weeks/months. This includes ensuring that the existing consumers work against the updates to the 0.8.2 consumer API as well as implementing support for the new consumer API introduced in 0.9. Roadmap information can be found here ( https://github.com/Parsely/pykafka/blob/master/doc/roadmap.r... ).

Re: Apache Kafka 0.9 is released

#17

Post by vdnkh »

simonw wrote: post #3

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

Same problem for .NET/C#. Nothing established/built enough to feel comfortable using it in production.

Re: Apache Kafka 0.9 is released

#18

Post by emmett9001 »

czinck wrote: post #10
simonw wrote: post #3

The worst thing about Kafka in my experience has been the consumer libraries for languages like Python. That's not to say that they are terrible or unusable, just that they don't have nearly as much polish as the core of Kafka itself. I'm very much looking forward to new client libraries built against the new consumer API.

I'd say the Python library I used was borderline unusable, we stopped using Kafka (it was just a trial period, wasn't rolled to production yet) because of limits in one of the most popular Python interfaces. The interface worked well enough, the API was good, but they didn't (and the bug tracker seemed to imply they wouldn't) support synchronizing reads across processes for the same group. What's the point in a distr…

Sounds like old news, but if this is still an issue, PyKafka does allow balanced reads across a consumer group. https://github.com/parsely/pykafka

Re: Apache Kafka 0.9 is released

#19

Post by zodvik »

mathnode wrote: post #7
erichmond wrote: post #4

I'm always fascinated by the lack of discussion around distributed systems tooling on HN. Anyway! Congrats!

The only other beast of similar nature that appears occasionally I can think of, is Onyx. Which seems pretty cool. Anyway,Kafka Connect will provide what probably most people are looking for in Samza.

We have built a fairly robust system along the idea of Kafka Connect and it's open sourced at https://github.com/flipkart/aesop. Currently, it supports MySQL & Hbase as sources and MySQL, Hbase, ES, Kafka as destination.

Post reply on HN