Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

161–170 of 229 posts

Re: Why was Apache Kafka created?

#161

Post by halifaxbeard »

HarHarVeryFunny wrote: post #143
Joeri wrote: post #37

Earlier quoted context omitted.

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

Why do you say log rather than just publish and subscribe?

the log stays on kafka for replay until your per log retention settings delete it

Re: Why was Apache Kafka created?

#162

Post by dbacar »

The biggest strength of Kafka in my opinion is consumer groups.I have been using it since 2016 in at least 3 projects and it never failed, not that big workloads though (~100 messages/sec max). However it is a bit difficult to monitor and manage using only the out of the box applications.

Re: Why was Apache Kafka created?

#163

Post by macmar »

In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational challenges.

Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:

Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.

Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.

Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.

The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.

From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.

Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.

Re: Why was Apache Kafka created?

#164

Post by grantedimmunity »

macmar wrote: post #163

In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational chall…

[dead]

Re: Why was Apache Kafka created?

#165

Post by nicoritschel »

dbacar wrote: post #162

The biggest strength of Kafka in my opinion is consumer groups.I have been using it since 2016 in at least 3 projects and it never failed, not that big workloads though (~100 messages/sec max). However it is a bit difficult to monitor and manage using only the out of the box applications.

I would take a hard look at Kinesis/Pubsub for these data volumes; should cost in the tens of dollars monthly.

Re: Why was Apache Kafka created?

#166

Post by fifilura »

physicles wrote: post #159
fifilura wrote: post #111

Earlier quoted context omitted.

I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well. The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate. Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it en…

Totally agree with this. I’ll add that replaying your data needs special tooling to 1) find the correct offsets on each topic, and 2) spin up whatever daemon will consume that data out-of-band from normal processing, and shut it down when completed. I don’t remember where I read this, but someone made the observation that writing a stream processing system is about 3x harder than writing a batch system, exactly for a…

Yes I'll happily extend that to 10x more difficult.

At least compared to building a batched pipeline with SQL. I think you should really think hard whether you really need a streaming pipeline. And even if you find that you do, it may be worthwhile to make a batched pipeline as your first implementation.

I did exactly what you describe in my previous job. In the beginning with reluctance from our architects who wanted to keep banging the dead horse and did not understand the power of SQL "SQL is not real programming, engineers write java" (ok maybe I deserve a straw-man yellow card here, they don't deserve all of that). But I think they understood after a while.

With AWS Athena and Airflow. Good luck, consider me your distant moral support.

Re: Why was Apache Kafka created?

#167

Post by supriyo-biswas »

macmar wrote: post #163

In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational chall…

This appears to be a LLM generated comment; if my assumption is correct - please do not do this here. Thank you.

Re: Why was Apache Kafka created?

#168

Post by HarHarVeryFunny »

HarHarVeryFunny wrote: post #143
Joeri wrote: post #37

Earlier quoted context omitted.

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

Why do you say log rather than just publish and subscribe?

The way people choose to use feedback on HN never fails to suprise me - we've got a generally intelligent user base here, but the most common type of feedback voting isn't because something is wrong but rather a childish "I don't like it - I want to suppress this comment".

In this case it's something different - this was an honest question, and received two useful replies, so why downvote?! The mental model of people using Kafka is useful to know - in this case the published data being more log-like than stream-like since it's retained per a TTL policy, with each "subscriber" having their own controllable read index.

Re: Why was Apache Kafka created?

#169

Post by slipperydippery »

gf000 wrote: post #112
slipperydippery wrote: post #76

Earlier quoted context omitted.

When all your examples in actual use are bloated… I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C…

Sounds like a classic case of confirmation bias. Especially that like half of the web runs on Java, you just have absolutely no idea when it silently does its job perfectly.

K.

Re: Why was Apache Kafka created?

#170

Post by munksbeer »

cowanon2222 wrote: post #155
munksbeer wrote: post #131

Earlier quoted context omitted.

> Or just freaking MQTT. Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work. We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many a…

I use MQTT daily. I'm not sure why the commenter suggested it; it is orthogonal to queueing or log streams. MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quick…

Thank you.
Post reply on HN