Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

31–40 of 229 posts

Re: Why was Apache Kafka created?

#31

Post by dijit »

varbhat wrote: post #18

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

I got really pissed off with their field CTO for essentially trying to pull the wool over my eyes regarding performance and reliability.

Essentially their base product (NATs) has a lot of performance but trades it off for reliability. So they add Jetstream to NATs to get reliability, but use the performance numbers of pure NATs.

I got burned by MongoDB for doing this to me, I won’t work with any technology that is marketed in such a disingenuous way again.

Re: Why was Apache Kafka created?

#32

Post by sea-gold »

varbhat wrote: post #18

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

There is a good comparison between NATS, Kakfa, and others here: https://docs.nats.io/nats-concepts/overview/compare-nats

Re: Why was Apache Kafka created?

#34

Post by slau »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

ZMQ is not a managed queue. It’s networking library.

Re: Why was Apache Kafka created?

#35

Post by rvz »

clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

> Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today.

Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.

Re: Why was Apache Kafka created?

#36

Post by tomrod »

majormajor wrote: post #14
zug_zug wrote: post #9

My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.

What off the shelf tools in 2012 would you propose, exactly?

Sounds like MQTT?

Re: Why was Apache Kafka created?

#37

Post by Joeri »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA.

If you need all those things, there just are not a lot of options.

Re: Why was Apache Kafka created?

#38

Post by rvz »

njitbew wrote: post #13
clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

> I'm curious, what exactly feels bloated about Java?

Everything.

Why do you think Kubernetes is NOT written in Java?

Re: Why was Apache Kafka created?

#39

Post by zug_zug »

majormajor wrote: post #14
zug_zug wrote: post #9

My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.

What off the shelf tools in 2012 would you propose, exactly?

Make it less event-orchestrated and use a db. It’s just a social network for recruiters it’s not as complicated as they like to pretend.

You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.

Re: Why was Apache Kafka created?

#40

Post by pojzon »

dktalks wrote: post #15

I worked on this while I was at LI and I think the major selling point back then was Replayability of messages but it was something similar to what you would get with Pub/Sub. We could also have multiple clients listening and processing same messages for their own purposes so you could use the same queue and have different clients process them as they wanted.

Its the ability to replay messages at later notice when needed.

At least this was the reason we decided to use Kafka instead of simple queues.

It was useful when we built new consumer types for the same data we already processed or we knew we gonna have later but cant build now due to prorities.

Post reply on HN