Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

111–120 of 229 posts

Re: Why was Apache Kafka created?

#111

Post by fifilura »

BiraIgnacio wrote: post #19

It was created to teach me the concept of love-hate relationships

I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well.

The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.

Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.

Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.

I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?

At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.

Re: Why was Apache Kafka created?

#112

Post by gf000 »

slipperydippery wrote: post #76
jayd16 wrote: post #53

Earlier quoted context omitted.

"Java is bloated because I only look at the bloated examples." Is C++ bloated because of the memory Chrome uses?

When all your examples in actual use are bloated… I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C…

Sounds like a classic case of confirmation bias.

Especially that like half of the web runs on Java, you just have absolutely no idea when it silently does its job perfectly.

Re: Why was Apache Kafka created?

#113

Post by EdwardDiego »

selkin wrote: post #84
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

KIP-932[0] adds queue semantics for Kafka. You may still want to use another queue though: as always, no one size fits all. [0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

I'm really unsure where the drive for this is coming from tbh. (cough CFLT share price since IPO, big enterprise customers cough) If this was so desirable, everyone would've jumped ship to Pulsar already.

Re: Why was Apache Kafka created?

#114

Post by diarrhea »

Joeri wrote: post #37
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

Perhaps the best terse summary of Kafka I have come across yet.

Re: Why was Apache Kafka created?

#115

Post by pdimitar »

atombender wrote: post #70
varbhat wrote: post #18

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

NATS is very good. It's important to distinguish between core NATS and Jetstream, however. Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No dur…

That's an amazing analysis, thank you so much!

What are your impressions of Red panda?

We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs.

Interested if Redpanda is doing better than either.

Re: Why was Apache Kafka created?

#116

Post by atombender »

pdimitar wrote: post #115
atombender wrote: post #70

Earlier quoted context omitted.

NATS is very good. It's important to distinguish between core NATS and Jetstream, however. Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No dur…

That's an amazing analysis, thank you so much! What are your impressions of Red panda? We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs. Interested if Redpanda is doing better than either.

Redpanda is fantastic, but it has the exact same message semantics as Kafka. They don't even have their own client; you connect using the Kafka protocol. Very happy with it, but it does have the same "whole batch or nothing" approach.

NATS/Jetstream is amazing if it fits your use case and you don't need extreme scalability. As I said before, it offers a lot more flexibility. You can process a stream sequentially but also nack messages, so you get the best of both worlds. It has deduping (new messages for the same subject will mark older ones as deleted) and lots of other convenience goodies.

Re: Why was Apache Kafka created?

#117

Post by kachapopopow »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

I do not recommend Redis (janky implementation, subscribers drop randomly, java libraries are a crime against humanity) or RabbitMQ (memory issues), ZMQ is not a messaging queue, named pipes are not reliable and what the hell is SQS.

Re: Why was Apache Kafka created?

#118

Post by pdimitar »

atombender wrote: post #116
pdimitar wrote: post #115

Earlier quoted context omitted.

That's an amazing analysis, thank you so much! What are your impressions of Red panda? We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs. Interested if Redpanda is doing better than either.

Redpanda is fantastic, but it has the exact same message semantics as Kafka. They don't even have their own client; you connect using the Kafka protocol. Very happy with it, but it does have the same "whole batch or nothing" approach. NATS/Jetstream is amazing if it fits your use case and you don't need extreme scalability. As I said before, it offers a lot more flexibility. You can process a stream sequentially but…

Thank you so much again. Yes, we are not Google scale, our main priority is durability and scalability but only up to a (I'd say fairly modest) point. I.e. be able to have one beefy NATS server do it all and only add a second one when things start getting bad. Even 3 servers we'd see as a strategic defeat + we have data but again, very far from Google scale.

We've looked at Redis streams but me and a few others are skeptical as Redis is not known for good durability practices (talking about the past; I've no idea if they pivoted well in the last years) and sadly none of us has any experience with MQTT -- though we heard tons of praise on that one.

But our story is: some tens of terabytes of data, no more than a few tens of millions of events / messages a day, aggressive folding of data in multiple relational DBs, and a very dynamic and DB-heavy UI (I will soon finish my ElixirRust SQLite3 wrapper so we're likely going to start sharding the DB-intensive customer data to separate SQLite3 databases and I'm looking forward to spearheading this effort; off-topic). For our needs NATS Jetstream sounds like the exactly perfect fit, though time will tell.

I still have the nagging feeling of missing out on still having not tried MQTT though...

Re: Why was Apache Kafka created?

#119

Post by edem »

EdwardDiego wrote: post #79
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?" The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better. Using S3 was better, but we still hit…

Do you have a blog somewhere? Where do I learn how to use Kafka properly? I like the idea behind it, but its quirks...not so much.

Re: Why was Apache Kafka created?

#120

Post by edem »

Joeri wrote: post #37
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

What do you think about Temporal?
Post reply on HN