Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

181–190 of 229 posts

Re: Why was Apache Kafka created?

#181

Post by enether »

djoldman wrote: post #154
enether wrote: post #151

Earlier quoted context omitted.

Where does this 17 PB/day number come from? I didn't quote any numbers directly. Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)

> In 2010, LinkedIn had 90 million members. Today, we serve over 1.2 billion members on LinkedIn. Unsurprisingly, this increase has created some challenges over the years, making it difficult to keep up with the rapid growth in the number, volume, and complexity of Kafka use cases. Supporting these use-cases meant running Kafka at a scale of over 32T records/day at 17 PB/day on 400K topics distributed across 10K+ mac…

~197 GB/s ... nice.

I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something.

Similarly, Uber has shared they push 89 GB/s through Kafka - 7.7 PB/s. People always ask me - what is a taxi/food-delivery app storing so much

Re: Why was Apache Kafka created?

#182

Post by HarHarVeryFunny »

MYEUHD wrote: post #177
HarHarVeryFunny wrote: post #168

Earlier quoted context omitted.

The way people choose to use feedback on HN never fails to suprise me - we've got a generally intelligent user base here, but the most common type of feedback voting isn't because something is wrong but rather a childish "I don't like it - I want to suppress this comment". In this case it's something different - this was an honest question, and received two useful replies, so why downvote?! The mental model of people…

> Please don't comment about the voting on comments. It never does any good, and it makes boring reading. https://news.ycombinator.com/newsguidelines.html#comments

Same goes for metacomments about voting.

Re: Why was Apache Kafka created?

#183

Post by alt227 »

enether wrote: post #181
djoldman wrote: post #154

Earlier quoted context omitted.

> In 2010, LinkedIn had 90 million members. Today, we serve over 1.2 billion members on LinkedIn. Unsurprisingly, this increase has created some challenges over the years, making it difficult to keep up with the rapid growth in the number, volume, and complexity of Kafka use cases. Supporting these use-cases meant running Kafka at a scale of over 32T records/day at 17 PB/day on 400K topics distributed across 10K+ mac…

~197 GB/s ... nice. I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something. Similarly, Uber has shared they push 89 GB/s through Kafka - 7.7 PB/s. People always ask me - what is a taxi/food-delivery app storing so much

The biggest capacity HD available today is 30TB.

17PB = about 567 of those drives... being totally filled... per day.

I was hoping somebody would come and say this is a simple spelling error or something.

The cost of the drives alone seems astronomical, let alone the logistics of the data center keeping up with storing that much data.

EDIT: I have just realised that they are probably only processing at this speed, rather than storing it, can anyone confirm if they store all the logs they process?

Re: Why was Apache Kafka created?

#184

Post by vlovich123 »

vips7L wrote: post #175
vlovich123 wrote: post #171

Earlier quoted context omitted.

That would be a compelling counter if and only if languages like Java actually beat other languages in throughput. In practice that doesn’t seem to be the case and the reasons for that seem to be: * languages like c++ and Rust simply don’t allocate as much as Java, instead using value types. Even C# is better here with value types being better integrated. * languages like c++ and Rust do not force atomic reference co…

> Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write). AFAIK that doesn’t really happen. They won’t insert atomic accesses anywhere on real hardware because the cpu is capable of doing that atomically anyway. > Additionally, a lot of Java code declares methods sync…

> Concurrency requires locks. Arc is a global lock on references

Concurrency does not require locks. There’s entire classes of lock free and wait free algorithms. Arc is also not a lock - it uses atomics to manage the reference counts and no operation on an Arc needs to wait on a lock (it is a lock-free container).

> “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.

Almost all objects that are used in a concurrent context will likely feature synchronized, at least historically. That’s why Hashtable was split into HashMap (unsynchronized) and ConcurrentHashMap (no longer using synchronized). Thats why you have StringBuffer which was redone into StringBuilder.

Re: Why was Apache Kafka created?

#185

Post by blast »

enether wrote: post #127
blast wrote: post #2

(I was wondering if this was some sort of generated ripoff but the author worked on Kafka for 6 years: https://x.com/BdKozlovski .)

What do you mean by "generated ripoff"? Are you saying it read like AI?

I mean that it looked like either AI generated or blogspam or both. Not the writing though, more the look of the page. Happy to be wrong.

Re: Why was Apache Kafka created?

#186

Post by dang »

macmar wrote: post #176
supriyo-biswas wrote: post #167

Earlier quoted context omitted.

This appears to be a LLM generated comment; if my assumption is correct - please do not do this here. Thank you.

No, man, this wasn't done by an LLM. I actually do Kafka implementations and follow exactly the same script I described. Perhaps the fact that I'm not a native English speaker may have caused this confusion. I just made sure my text was written correctly in English with a translator.

Non-native English speakers are 100% welcome on Hacker News, and the rest of us are amazed by how good your English usually is.

Generated comments are what people here feel strongly about rejecting. Not just LLM-written ones, but also comments that have been filtered through translators/editors/checkers and so on.

Readers are becoming hypersensitive to these, so it is better to just write in your own voice and post that. The benefit of authenticity outweighs the cost of a few grammar or spelling errors.

Re: Why was Apache Kafka created?

#187

Post by alt227 »

gf000 wrote: post #112
slipperydippery wrote: post #76

Earlier quoted context omitted.

When all your examples in actual use are bloated… I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C…

Sounds like a classic case of confirmation bias. Especially that like half of the web runs on Java, you just have absolutely no idea when it silently does its job perfectly.

> Especially that like half of the web runs on Java

Source?

W3 seems to think its more like ~5%

https://w3techs.com/technologies/comparison/pl-java

Re: Why was Apache Kafka created?

#188

Post by alt227 »

enether wrote: post #151
alt227 wrote: post #138

> 17 PB/day Are they really generating data at this scale? I cant even imagine a system which creates and stores this much data this fast.

Where does this 17 PB/day number come from? I didn't quote any numbers directly. Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)

Sorry I meant to reply to this post above, who linked the blog post which mentions the 17PB/S

Post: https://news.ycombinator.com/item?id=45002089

Re: Why was Apache Kafka created?

#189

Post by enether »

blast wrote: post #185
enether wrote: post #127

Earlier quoted context omitted.

What do you mean by "generated ripoff"? Are you saying it read like AI?

I mean that it looked like either AI generated or blogspam or both. Not the writing though, more the look of the page. Happy to be wrong.

It's neither. Only the thumbnail background is AI generated.

The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me.

I'm simply curious what parts give that "cheap" look so I can improve. On Reddit I've had massive amounts of downvotes because they guess the content is AI, when in fact no AI is used in the creation process at all.

One guess I have is the bullet points + bolding combo. Most AIs use a ton of that, and rightly so, because it aids in readability.

Re: Why was Apache Kafka created?

#190

Post by vips7L »

vlovich123 wrote: post #184
vips7L wrote: post #175

Earlier quoted context omitted.

> Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write). AFAIK that doesn’t really happen. They won’t insert atomic accesses anywhere on real hardware because the cpu is capable of doing that atomically anyway. > Additionally, a lot of Java code declares methods sync…

> Concurrency requires locks. Arc is a global lock on references Concurrency does not require locks. There’s entire classes of lock free and wait free algorithms. Arc is also not a lock - it uses atomics to manage the reference counts and no operation on an Arc needs to wait on a lock (it is a lock-free container). > “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t. Almost all ob…

Ok I mispoke on Arc because I was being hasty; but you're still being pedantic. Concurrency still requires locks. Wait/lock free algorithms can't cover the entirety of concurrency. Rust ships with plenty of locks in std::sync and to implement a ConcurrentHashMap in Rust you would still need to lock. In fact it doesn't even look like Rust supplies concurrent collections at all. So what are we even talking about here? This is still a far cry from "a lot of Java objects use global synchronized locks".
Post reply on HN