Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

131–140 of 229 posts

Re: Why was Apache Kafka created?

#131

Post by munksbeer »

adev_ wrote: post #27
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

> Why they'd need Kafka was always our first question, never got a good answer from a single one of them "To follow the hype train, Bro" is often the real answer. > If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. Or just freaking MQTT. MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messag…

> Or just freaking MQTT.

Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.

We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.

So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?

Re: Why was Apache Kafka created?

#132

Post by munksbeer »

AtlasBarfed wrote: post #52
dijit wrote: post #31

Earlier quoted context omitted.

I got really pissed off with their field CTO for essentially trying to pull the wool over my eyes regarding performance and reliability. Essentially their base product (NATs) has a lot of performance but trades it off for reliability. So they add Jetstream to NATs to get reliability, but use the performance numbers of pure NATs. I got burned by MongoDB for doing this to me, I won’t work with any technology that is ma…

Don't implement any distributive technology until aphyr has put it through the paces, and even then... Pilot

https://aphyr.com/about

"Unavailable Due to the UK Online Safety Act"

:(

Re: Why was Apache Kafka created?

#133

Post by pm90 »

kafcausir wrote: post #130
pss314 wrote: post #104

LinkedIn recently announced that it transitioned from Kafka to Northguard. Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2] [1]: https://www.linkedin.com/blog/engineering/infrastructure/int... [2]: https://www.youtube.com/watch?v=RDV6-MUVEbQ

I haven't been following kafka for some years now, but i thought Linkedin were heavily invested in it. What happened? Also what happened to Confluent? Their team were ex-Linkedin members from what i remember?

Confluent is expensive and I don’t believe LI used them; they did use OSS kafka. Im guessing that after being acquired by MS they explored other tech.

Re: Why was Apache Kafka created?

#134

Post by munksbeer »

von_lohengramm wrote: post #47
buckle8017 wrote: post #28

Earlier quoted context omitted.

Java the language and Java the runtime are fine. The way most Java code is written is terrible Enterprise factory factory factory.

The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all? Just find some other language that targets the JVM. But then you're already treading such DIY and frictionful waters that just adopting some other cross-platform language/runtime isn't the worst idea.

>The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all?

The reason is quite well known. Supporting multiple languages is a cost. If you only have to support one language, everything is simpler and cheaper.

With Java, you can write elegant code these days, rely on ZGC, not really worry too much about GC and get excellent performance with quick development cycles for most of your use cases. Then with the same language and often in the same repo (monorepo is great) you can write smarter code for your hot path in a GC free manner and get phenomenal performance.

And you get that with only having one build system, one CI pipeline, one deployment system, some amazing profiling and monitoring tooling, a bunch of shared utility code that you don't have to duplicate, and a lot more benefits.

That's the reason to choose Java.

Of course, if you're truly into HFT space, then they'll be writing in C, C++ or on FPGAs.

Re: Why was Apache Kafka created?

#135

Post by alt227 »

mldbk wrote: post #97
slipperydippery wrote: post #16

Earlier quoted context omitted.

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…

I held the same view as you when I was 22, more than 15 years ago. With over 15 years of professional experience since then, my perspective has shifted: Java demonstrates its strength when stability, performance, and scalability are required (e.g. bloody enterprise) A common misconception comes from superficial benchmarking. Many focus solely on memory consumption, which often provides a distorted picture of actual s…

> Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.

When other languages can do the same thing with an order of magnitude less RAM, any other efficencies in the system tend to be overshadowed by that and be the sticking point in peoples memories.

You may argue that holding on to this extra memory makes subsequent calls and reads quicker etc, but in my experience generally people are willing to sacrifice milliseconds to gain gigabytes of memory.

Re: Why was Apache Kafka created?

#136

Post by zozbot234 »

gf000 wrote: post #106
vlovich123 wrote: post #98

Earlier quoted context omitted.

No but it does speak to the memory overhead of tracing GC vs ref counting as garbage collection strategies.

Which is very important in... embedded settings. While for typical backend situations, reference counting has a crazy high throughput overhead, doing atomic inc/decs left and right, that instantly trashes any kind of cache, and does it in the mutator thread that would do the actual work, for the negligible benefit of using less memory. Meanwhile a tracing GC can do (almost) all its work in another thread, not slowing…

It depends how you implement reference counting. In Rust the atomic inc-dec operations can be kept at a minimum (i.e. only for true changes in lifecycle ownership) because most accesses are validated at compile time by the borrow checker.

Re: Why was Apache Kafka created?

#137

Post by alt227 »

throw310822 wrote: post #126
theyinwhy wrote: post #62

Earlier quoted context omitted.

It is called Kafka because it can write.

It's a bit like calling a dictation software "Hitler" because he also liked to dictate.

Thats a brilliant idea, although if I ever create a dictation software I am going to call it 'Mussolini'.

Re: Why was Apache Kafka created?

#139

Post by throw310822 »

alt227 wrote: post #137
throw310822 wrote: post #126

Earlier quoted context omitted.

It's a bit like calling a dictation software "Hitler" because he also liked to dictate.

Thats a brilliant idea, although if I ever create a dictation software I am going to call it 'Mussolini'.

> Thats a brilliant idea

I know, Claude was also enthusiastic about it.

Re: Why was Apache Kafka created?

#140

Post by gdbsjjdn »

kafcausir wrote: post #130
pss314 wrote: post #104

LinkedIn recently announced that it transitioned from Kafka to Northguard. Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2] [1]: https://www.linkedin.com/blog/engineering/infrastructure/int... [2]: https://www.youtube.com/watch?v=RDV6-MUVEbQ

I haven't been following kafka for some years now, but i thought Linkedin were heavily invested in it. What happened? Also what happened to Confluent? Their team were ex-Linkedin members from what i remember?

NorthGuard looks like a clean sheet redesign of Kafka. The OSS Kafka community has taken a long time to implement things like KRaft, which addressed metadata scalability concerns by storing metadata in the brokers themselves (it used to be in a separate data store called ZooKeeper which was operationally complicated). NorthGuard also supports splitting and merging ranges of keys without repartitioning the entire existing dataset. The way records are assigned to partitions is a big problem in running Apache Kafka at scale because it requires predicting the key distribution and number of partitions ahead of time.

Confluent is 11 years old and IPOed several years ago. It was founded by 3 ex-LinkedIn people who originally designed Kafka. 2 of the 3 founders are still at the company. LinkedIn never used Confluent, Confluent was a company founded to sell an enterprise version of the open source project (and later a cloud version).

Post reply on HN