Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Kafka 0.9 is released

confluent.io

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Apache Kafka 0.9 is released

#21

Post by ah- »

nehanarkhede wrote: post #9
felipesabino wrote: post #5

I've been using kafka 0.8.2 for some time now together with Node.js for both consumer and producer. Although the producer side is quite simple to use and have more than one option available, the consumer side there is only one project that is "maintained" and works [1][2], all other opstions either only have producer available [3] or have not received a commit in years [4]. I am a bit disappointed about how little at…

Your critique is well received. The Apache Kafka project has support for the Java clients and the non-java clients will be developed and available in a federated manner. At Confluent, we are focused on providing first class non-java clients that are API and functionality compatible with the java clients. Forthcoming releases of the Confluent Platform will include a C/C++, python and node.js client. Stay tuned http://…

The C/C++ client, will that be librdkafka? If I need a high-quality client for another language, would you recommend building that using librdkafka or the rest proxy?

Re: Apache Kafka 0.9 is released

#23

Post by btown »

mixmastamyk wrote: post #20

What is the use case for this product? Could you use it as a replicating database across sites?

Kafka is a replicating database, but not of randomly accessible or queryable data - rather, it's for logs where you can start at any point in time (including realtime), and play the log forwards from there. If you aggregate access logs into it, then use it to feed stream processing frameworks (or feed it into Hadoop for bulk processing), you can use it very effectively for analytics workloads.

Or you can run your entire business around events-as-ground-truth rather than SQL-style-domain-tables-as-ground-truth. Various modules consume from Kafka and write aggregated records back into Kafka - basically a directed graph of processing modules with Kafka to implement every edge - and finally there's a module that translates the final product into what is essentially a live-updating materialized view for your web backend to consume. LinkedIn does exactly this - they open-sourced Kafka and spun out Confluent to help others use this model.

Stream processing is a very powerful way of thinking about data management - with the great side effect that "migrations" of data tables are limited by your imagination, and they never run the risk of data corruption. We use this paradigm, though not Kafka itself (Mongo supports this paradigm and simplifies a lot of things if you aren't yet at LinkedIn scale), in production at http://belstone.com.

Martin Kleppman's talks are a great place to start if you want to learn more: http://www.confluent.io/blog/making-sense-of-stream-processi... is an excellent overview.

Re: Apache Kafka 0.9 is released

#24

Post by SEJeff »

nehanarkhede wrote: post #9
felipesabino wrote: post #5

I've been using kafka 0.8.2 for some time now together with Node.js for both consumer and producer. Although the producer side is quite simple to use and have more than one option available, the consumer side there is only one project that is "maintained" and works [1][2], all other opstions either only have producer available [3] or have not received a commit in years [4]. I am a bit disappointed about how little at…

Your critique is well received. The Apache Kafka project has support for the Java clients and the non-java clients will be developed and available in a federated manner. At Confluent, we are focused on providing first class non-java clients that are API and functionality compatible with the java clients. Forthcoming releases of the Confluent Platform will include a C/C++, python and node.js client. Stay tuned http://…

For those that don't realize it, Neha and Jay were two of the main developers who wrote Kafka.

Thanks for the heads up!

Re: Apache Kafka 0.9 is released

#26

Post by pcsanwald »

Were the diagrams done with software, or hand drawn? If software, I'm curious what package/style you used, the style looks very similar to Martin Kleppmann's presentation at StrangeLoop; I assumed his were hand drawn but I'm realizing now this might be a omni style or something.

Re: Apache Kafka 0.9 is released

#30

Post by optimusclimb »

pcsanwald wrote: post #26

Were the diagrams done with software, or hand drawn? If software, I'm curious what package/style you used, the style looks very similar to Martin Kleppmann's presentation at StrangeLoop; I assumed his were hand drawn but I'm realizing now this might be a omni style or something.

This always comes up with Martin Kleppman diagrams. See this discussion:

https://news.ycombinator.com/item?id=9613118

The bottom comments seem to agree it was done using Paper.

Post reply on HN