Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

201–210 of 229 posts

Re: Why was Apache Kafka created?

#201

Post by lossolo »

enether wrote: post #198
alt227 wrote: post #183

Earlier quoted context omitted.

The biggest capacity HD available today is 30TB. 17PB = about 567 of those drives... being totally filled... per day. I was hoping somebody would come and say this is a simple spelling error or something. The cost of the drives alone seems astronomical, let alone the logistics of the data center keeping up with storing that much data. EDIT: I have just realised that they are probably only processing at this speed, ra…

I would assume storage varies greatly. I know that LinkedIn quoted an average read fanout ratio of 5.5x in Kafka, meaning each byte was read 5.5x times. Assuming that is still true, we ought to divide by 6.5x to get to the daily write amount That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replicatio…

> That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)

Yep. Whole week can be easily stored in 1-2 racks.

Re: Why was Apache Kafka created?

#202

Post by laughing_man »

adev_ wrote: post #197
laughing_man wrote: post #94

Earlier quoted context omitted.

That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.

> That decision cost the company millions. And 5 years later the responsible of the decision left the company with a giant pile of mess behind him/her. But let's see things positively: he can now add "Kafka at scale" on the CV.

That is exactly what happened.

Re: Why was Apache Kafka created?

#203

Post by vlovich123 »

gf000 wrote: post #194
vlovich123 wrote: post #171

Earlier quoted context omitted.

That would be a compelling counter if and only if languages like Java actually beat other languages in throughput. In practice that doesn’t seem to be the case and the reasons for that seem to be: * languages like c++ and Rust simply don’t allocate as much as Java, instead using value types. Even C# is better here with value types being better integrated. * languages like c++ and Rust do not force atomic reference co…

The main topic here was Swift vs Android's Java. Of course with manual memory management you may be able to write more efficient programs, though it is not a given, and comes at the price of a more complicated and less flexible programming model. At least with Rust, it is actually memory safe, unlike c++. - ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to u…

> ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.

That’s a bold claim to make that doesn’t seem to actually be true from my experience. Your 5ghz CPU can probably do ~20 billion non atomic reference adjustments whereas your GC system has to have atomics all over the place or it won’t work and atomics have parasitic performance on unrelated code due to bus locks and whatnot.

> Java can also do on-stack replacement.. sometimes

That’s not what this is. It’s called hybrid RC and it applies always provided you follow the rules.

> The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.

The argument here is not about developer productivity - the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++. That just isn’t true so you end up sacrificing throughput AND latency AND peak memory usage. You may not care and are fine with that tradeoff, but claiming you’re not making that tradeoff is not based on the facts.

Re: Why was Apache Kafka created?

#204

Post by atombender »

erulabs wrote: post #67

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was…

For a long time I've wondered if we could just invent an extension for Postgres that allow physically ordered, append-only tables.

The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.

Kafka is of course more than that, but I bet you can get quite far with this.

Re: Why was Apache Kafka created?

#205

Post by vlovich123 »

vips7L wrote: post #190
vlovich123 wrote: post #184

Earlier quoted context omitted.

> Concurrency requires locks. Arc is a global lock on references Concurrency does not require locks. There’s entire classes of lock free and wait free algorithms. Arc is also not a lock - it uses atomics to manage the reference counts and no operation on an Arc needs to wait on a lock (it is a lock-free container). > “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t. Almost all ob…

Ok I mispoke on Arc because I was being hasty; but you're still being pedantic. Concurrency still requires locks. Wait/lock free algorithms can't cover the entirety of concurrency. Rust ships with plenty of locks in std::sync and to implement a ConcurrentHashMap in Rust you would still need to lock. In fact it doesn't even look like Rust supplies concurrent collections at all. So what are we even talking about here?…

No, that’s an overly strong statement - concurrency doesn’t necessarily require locks even though they can be convenient to express it. You could have channels and queues to transfer data and ownership between threads. Not a lock in sight as queues and channels can be done lock free. The presence of locks in the Rust standard library says nothing other than it’s a very common concurrency tool, not that it’s absolutely required.

> and to implement a ConcurrentHashMap in Rust you would still need to lock

There’s many ways to implement concurrency safe hashmaps (if you explicitly needs such a data structure as the synchronization mechanism) without locks. Notably RCU is such a mechanism (really neat mechanism developed for the kernel although not super user friendly yet or common in userspace) and there are also generational garbage techniques available (kind of similar to tracing GC conceptually but implemented just for a single data structure). A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.

Re: Why was Apache Kafka created?

#206

Post by prerok »

stickfigure wrote: post #199
prerok wrote: post #173

Earlier quoted context omitted.

SQS is simple queueing service in AWS. It's ok, guarantees at least one time delivery, but I am not sure how useful it is for large volumes of messages (by this I just mean that we use it for low volume messages and I don't have experience when using larger volumes).

SQS is fantastic at exceptionally high total volumes of messages - you probably can't saturate it. But it's not great for streaming a list of ordered messages. SQS has a FIFO mode but performance will never be what you can get out of Kafka. Also, SQS isn't pub/sub. Kafka and SQS really have very different use cases.

Agreed, I was just trying to answer the parent's question as to what it is.

Re: Why was Apache Kafka created?

#207

Post by gf000 »

vlovich123 wrote: post #203
gf000 wrote: post #194

Earlier quoted context omitted.

The main topic here was Swift vs Android's Java. Of course with manual memory management you may be able to write more efficient programs, though it is not a given, and comes at the price of a more complicated and less flexible programming model. At least with Rust, it is actually memory safe, unlike c++. - ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to u…

> ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs. That’s a bold claim to make that doesn’t seem to actually be true from my experience. Your 5ghz CPU can probably do ~20 billion non atomic referen…

> the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++

No, that has never been the specific claim - you can always write more efficient code with manual memory management, given enough time, effort and skill. I wasn't even the one who brought up c++ and rust. Like literally I write this twice in my comment.

What I'm talking about is reference counting as a GC technique vs tracing as a GC technique, all else being equal - it would be idiotic to compare these two if no other "variable" is fixed. (Oh and I didn't even mention the circular references problem, which means you basically have to add a tracing step either-way unless you restrict your language so that it can't express circular stuff).

As for the atomic part, sure, if all it would do is non-atomic increments then CPUs would be plenty happy. And you are right that depending on how the tracing GC is implemented, it will have a few atomic instructions. What you may miss is how often each run. On almost every access, vs every once in a while on a human timescale. Your OS scheduler will also occasionally trash the performance of your thread. But this is the actually apples to oranges comparison, and both techniques can do plenty of tweaks to hide certain tradeoffs, at the price of something else.

And I also mention that the above triad of time, skill and effort is not a given and is definitely not free.

Re: Why was Apache Kafka created?

#208

Post by GoblinSlayer »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

I think our salesmen were happier to sell kafka, because it's enterprisey. Redis is better? Well, now we use kafka and redis.

Re: Why was Apache Kafka created?

#209

Post by 62951413 »

chamomeal wrote: post #96
throwaway8242 wrote: post #75

Earlier quoted context omitted.

That doesn't match my experience in the last 15 years working for 3 companies (one was a big enterprise, one medium sized and one startup) Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.

Have you gotten to use loom/virtual threads? I’ve heard pretty interesting stuff about em, but haven’t really spent the time to get into it yet. It’s pretty exciting and tbh gives me an easy elevator pitch to JVM world for people outside of it

If you have a use-case where you currently allocate ~1K threads mostly waiting on I/O switching to virtual threads is a one-liner ("Thread.ofVirtual()" instead of "Thread.ofPlatform()"). No more golang envy for sure.

Depending on how much memory is used by the Thread stack (presumably 1M-512K by default, allegedly 128K with Alpine base images) that's your 1G-500M heap space usage improvement right off the bat.

The migration from JDK17 to JDK21 was uneventful in production. The only issue is limited monitoring as a thread dump will not show most virtual threads and the micrometer metrics will not even collect the total number of active virtual threads. It's supposed to work better in JDK24.

The Spring Framework directly supports virtual threads with "spring.threads.virtual.enabled=true" but I haven't tried it to comment.

Re: Why was Apache Kafka created?

#210

Post by 62951413 »

mdaniel wrote: post #91
AtlasBarfed wrote: post #58

Earlier quoted context omitted.

I thought Kafka ditched the zookeeper

I wish; KIP-500[1] was the "let's use native Raft" implementation but I have never once in my life seen anyone using Kafka in KIP-500 mode 1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...

Very few people want to re-create their production AWS MSK cluster from scratch. And that's the only way currently. MSK can usually upgrade Kafka brokers with minor performance degradation but not for this particular change.
Post reply on HN