Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

211–220 of 229 posts

Re: Why was Apache Kafka created?

#211

Post by alt227 »

gf000 wrote: post #196
alt227 wrote: post #187

Earlier quoted context omitted.

> Especially that like half of the web runs on Java Source? W3 seems to think its more like ~5% https://w3techs.com/technologies/comparison/pl-java

5% of "whose server-side programming language we know" From the website. And 76% of these websites is PHP, which seems to mean.. they can determine PHP more easily for a website (nonetheless, there are indeed a lot of WordPress sites, but not this amount).

Right, so Im assuming that as you are saying 'Half the web runs on java', maybe you know more about what websites are using in their backend? Care to share where you are getting this information from?

Re: Why was Apache Kafka created?

#212

Post by vips7L »

vlovich123 wrote: post #205
vips7L wrote: post #190

Earlier quoted context omitted.

Ok I mispoke on Arc because I was being hasty; but you're still being pedantic. Concurrency still requires locks. Wait/lock free algorithms can't cover the entirety of concurrency. Rust ships with plenty of locks in std::sync and to implement a ConcurrentHashMap in Rust you would still need to lock. In fact it doesn't even look like Rust supplies concurrent collections at all. So what are we even talking about here?…

No, that’s an overly strong statement - concurrency doesn’t necessarily require locks even though they can be convenient to express it. You could have channels and queues to transfer data and ownership between threads. Not a lock in sight as queues and channels can be done lock free. The presence of locks in the Rust standard library says nothing other than it’s a very common concurrency tool, not that it’s absolutel…

> A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.

Still not in the standard library. The only way in Rust is to use a global lock around map. Seems to be worse than the situation in Java. You could implement the same thing and use a third party library in Java too. So your original point of "everything uses a global lock" is "overly strong"

Re: Why was Apache Kafka created?

#213

Post by vlovich123 »

gf000 wrote: post #207
vlovich123 wrote: post #203

Earlier quoted context omitted.

> ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs. That’s a bold claim to make that doesn’t seem to actually be true from my experience. Your 5ghz CPU can probably do ~20 billion non atomic referen…

> the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++ No, that has never been the specific claim - you can always write more efficient code with manual memory management, given enough time, effort and skill . I wasn't even the one who brought up c++ and rust. Like literally I write this twice in my comment. What I'm talking about is reference counting as a…

In Rust there’s no forcing of any specific garbage collection mechanism. You’re free to do whatever and there’s many high performance crates to let you accomplish this. Even in Swift this is opt-in.

As for “skill” this is one thing that’s super hard to optimize for. All I can do is point to existence proofs that there’s no mainstream operating system, browser or other piece of high performance code written in Java and it’s all primarily C/C++ with some assembly with Rust starting to take over the C/C++ bits. And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.

Re: Why was Apache Kafka created?

#214

Post by SJC_Hacker »

HarHarVeryFunny wrote: post #145
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case. I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a perfo…

Out of curiousity what was the issue with just wrapping the AMQP-CPP pub/sub calls around a mutex?

Re: Why was Apache Kafka created?

#215

Post by vlovich123 »

vips7L wrote: post #212
vlovich123 wrote: post #205

Earlier quoted context omitted.

No, that’s an overly strong statement - concurrency doesn’t necessarily require locks even though they can be convenient to express it. You could have channels and queues to transfer data and ownership between threads. Not a lock in sight as queues and channels can be done lock free. The presence of locks in the Rust standard library says nothing other than it’s a very common concurrency tool, not that it’s absolutel…

> A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap. Still not in the standard library. The only way in Rust is to use a global lock around map. Seems to be worse than the situation in Java. You could implement the same thing and use a third party library in Java too. So your original point of "everything uses a global lock" is "overly strong"

You’ve now degraded the conversation into a very very weird direction. You made a claim that concurrency required locks. It simply does not and I have an existence proof of Dashmap as a hashmap that doesn’t have any locks anywhere.

The strengths and weaknesses of the standard library aren’t relevant. But if we’re going there, the reason they’re not in the Rust standard library is likely in practice concurrent data structures are an anti pattern - putting a lock around a data structure doesn’t suddenly solve higher order race conditions which is something a lot of Java programmers seem to believe because the standard library encourages this kind of thinking.

As for “my comment” about “global lock” (your words not mine), it’s that the implicit lock that’s available on every object is a bad idea for highly concurrent code (not to mention the implicit overhead that implies for every part of the object graph regardless of it being needed anywhere). Don’t get me wrong - Java took a valiant effort to define a solid memory model for concurrency when the field was still young. Many of the ideas didn’t pan out and are antipatterns these days for high performing code. Of course none of that pertains to the original point of the conversation - tracing GCs have significantly more overhead in practice because they’re very difficult to be opt in, carry quite a penalty if not, Rc/Arc is much better as it’s possible to do opt-in when you need shared ownership (which isn’t always), and in practice loops don’t come up often enough to matter and when they do there’s still solutions. In other words tracing GCs drop huge amounts of performance on the floor and you can read all the comments to see how the claims are “it’s more efficient than Rc”, or “performance is free” or even “it doesn’t matter because the programmer is more efficient”. I’d buy the efficiency argument when the only alternative was C/C++ and came with serious memory safety baggage, but not any of the others and memory safety without sacrificing performance of C++ in my view is a solved problem with Rust.

Re: Why was Apache Kafka created?

#216

Post by gf000 »

vlovich123 wrote: post #213
gf000 wrote: post #207

Earlier quoted context omitted.

> the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++ No, that has never been the specific claim - you can always write more efficient code with manual memory management, given enough time, effort and skill . I wasn't even the one who brought up c++ and rust. Like literally I write this twice in my comment. What I'm talking about is reference counting as a…

In Rust there’s no forcing of any specific garbage collection mechanism. You’re free to do whatever and there’s many high performance crates to let you accomplish this. Even in Swift this is opt-in. As for “skill” this is one thing that’s super hard to optimize for. All I can do is point to existence proofs that there’s no mainstream operating system, browser or other piece of high performance code written in Java an…

Sure, but I think that people often fall into the trap of imagining a problem that nicely fits a RAII model, where each lifetime is statically knowable. This is either due to having a specific problem, or because we decided on a specific constraint.

Java is used in HFT (well, there are two types of "high frequency", one where general purpose CPUs are already too slow, where it obviously doesn't apply (neither do rust or c++)) - but sure, I wouldn't write a runtime or other piece of code in Java where absolute control over the hardware is required. But that's a small niche only. What about large distributed systems/algorithms? Why is Java over-represented in this niche (e.g. Kafka, Elasticsearch, etc)?

> And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.

That's subjective.

Re: Why was Apache Kafka created?

#217

Post by Gee101 »

EdwardDiego wrote: post #79
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?" The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better. Using S3 was better, but we still hit…

Have you found a good Head for Kafka to easily query the Topics using a SQL like language? Especially something that can infer table schema from the Schema Registry.

Re: Why was Apache Kafka created?

#218

Post by HarHarVeryFunny »

SJC_Hacker wrote: post #214
HarHarVeryFunny wrote: post #145

Earlier quoted context omitted.

IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case. I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a perfo…

Out of curiousity what was the issue with just wrapping the AMQP-CPP pub/sub calls around a mutex?

I'm a bit hazy on the full details because it was a few years ago, but basically it gets more complicated because you subscribe by installing an async callback which needs to ack/nak messages, needing locking, and will be called from the context of the network event loop that also needs locking. If you do any real work in the message processing callback then you'll be blocking the event loop, so the callback has to defer processing by queuing C++ lambdas capturing the context, and running those in a thread pool.

Re: Why was Apache Kafka created?

#219

Post by iamcreasy »

Joeri wrote: post #37
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

Why the delineation? As long as you have a queue - you can publishing whatever you want in it, right? Be it log or something else.

Re: Why was Apache Kafka created?

#220

Post by EdwardDiego »

Gee101 wrote: post #217
EdwardDiego wrote: post #79

Earlier quoted context omitted.

I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?" The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better. Using S3 was better, but we still hit…

Have you found a good Head for Kafka to easily query the Topics using a SQL like language? Especially something that can infer table schema from the Schema Registry.

KSQL?
Post reply on HN