Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

191–200 of 229 posts

Re: Why was Apache Kafka created?

#191
post #13
post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Have you ever run on-prem Atlassian products or any enterprise JVM apps?

They hog RAM, are slow, and are a bitch to configure.

Re: Why was Apache Kafka created?

#193
post #185

Earlier quoted context omitted.

I mean that it looked like either AI generated or blogspam or both. Not the writing though, more the look of the page. Happy to be wrong.

It's neither. Only the thumbnail background is AI generated. The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me. I'm simply curious what parts give that "cheap" look so I can improve. On Reddit I've had massive amounts of downvotes because they guess the content is AI, when in fact no AI is used in the creation process at all. One guess I have is th…

[deleted]

Re: Why was Apache Kafka created?

#194
post #106

Earlier quoted context omitted.

Which is very important in... embedded settings. While for typical backend situations, reference counting has a crazy high throughput overhead, doing atomic inc/decs left and right, that instantly trashes any kind of cache, and does it in the mutator thread that would do the actual work, for the negligible benefit of using less memory. Meanwhile a tracing GC can do (almost) all its work in another thread, not slowing…

That would be a compelling counter if and only if languages like Java actually beat other languages in throughput. In practice that doesn’t seem to be the case and the reasons for that seem to be: * languages like c++ and Rust simply don’t allocate as much as Java, instead using value types. Even C# is better here with value types being better integrated. * languages like c++ and Rust do not force atomic reference co…

The main topic here was Swift vs Android's Java.

Of course with manual memory management you may be able to write more efficient programs, though it is not a given, and comes at the price of a more complicated and less flexible programming model. At least with Rust, it is actually memory safe, unlike c++.

- ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.

> You also only need to atomic increment / decrement when ownership is being transferred to a thread

Java can also do on-stack replacement.. sometimes.

- regarding lower level capabilities, java does have an experimental Vector API for simd. Atomics are readily available in the language.

- Java's memory model only requires 32-bit writes to be "atomic" (though in actuality the only requirement is to not tear - there is no happens before relation in the general case, and that's what is expensive), though in practice 64-bit is also atomic, both of which are free on modern hardware. Field acces is not different from what rust or c++ does, AFAIK in the general case. And `synchronized` is only used when needed - it's just syntactic convenience. This depends on the algorithm at hand, there is no difference between the same algorithm written in rust/c++ vs java from this perspective. If it's lockless, it will be lockless in Java as well. If it's not, than all of them will have to add a lock.

The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.

Re: Why was Apache Kafka created?

#195

Earlier quoted context omitted.

> Or just freaking MQTT. Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work. We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many a…

I use MQTT daily. I'm not sure why the commenter suggested it; it is orthogonal to queueing or log streams. MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quick…

> it is orthogonal to queueing or log streams.

That is currently the problem.

A lot of usage of Kafka I have seen in the wild are not for log stream or queing but deployed as a simple pub/sub messaging service because "why not".

Re: Why was Apache Kafka created?

#196
post #187
post #112

Earlier quoted context omitted.

Sounds like a classic case of confirmation bias. Especially that like half of the web runs on Java, you just have absolutely no idea when it silently does its job perfectly.

> Especially that like half of the web runs on Java Source? W3 seems to think its more like ~5% https://w3techs.com/technologies/comparison/pl-java

5% of "whose server-side programming language we know"

From the website.

And 76% of these websites is PHP, which seems to mean.. they can determine PHP more easily for a website (nonetheless, there are indeed a lot of WordPress sites, but not this amount).

Re: Why was Apache Kafka created?

#197

Earlier quoted context omitted.

It's resume-driven development. It honestly can make sense for both company and employee. Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle. And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack over…

That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.

> That decision cost the company millions.

And 5 years later the responsible of the decision left the company with a giant pile of mess behind him/her.

But let's see things positively: he can now add "Kafka at scale" on the CV.

Re: Why was Apache Kafka created?

#198
post #183

Earlier quoted context omitted.

~197 GB/s ... nice. I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something. Similarly, Uber has shared they push 89 GB/s through Kafka - 7.7 PB/s. People always ask me - what is a taxi/food-delivery app storing so much

The biggest capacity HD available today is 30TB. 17PB = about 567 of those drives... being totally filled... per day. I was hoping somebody would come and say this is a simple spelling error or something. The cost of the drives alone seems astronomical, let alone the logistics of the data center keeping up with storing that much data. EDIT: I have just realised that they are probably only processing at this speed, ra…

I would assume storage varies greatly. I know that LinkedIn quoted an average read fanout ratio of 5.5x in Kafka, meaning each byte was read 5.5x times. Assuming that is still true, we ought to divide by 6.5x to get to the daily write amount

That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)

Re: Why was Apache Kafka created?

#199
post #173

Earlier quoted context omitted.

I do not recommend Redis (janky implementation, subscribers drop randomly, java libraries are a crime against humanity) or RabbitMQ (memory issues), ZMQ is not a messaging queue, named pipes are not reliable and what the hell is SQS.

SQS is simple queueing service in AWS. It's ok, guarantees at least one time delivery, but I am not sure how useful it is for large volumes of messages (by this I just mean that we use it for low volume messages and I don't have experience when using larger volumes).

SQS is fantastic at exceptionally high total volumes of messages - you probably can't saturate it. But it's not great for streaming a list of ordered messages. SQS has a FIFO mode but performance will never be what you can get out of Kafka.

Also, SQS isn't pub/sub. Kafka and SQS really have very different use cases.

Re: Why was Apache Kafka created?

#200
post #67

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was…

> that SQL databases are _not well suited_ to drinking from a firehose of writes

Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…

Post reply on HN