Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

71–80 of 229 posts

Re: Why was Apache Kafka created?

#71

Post by zaphirplane »

jeremyjh wrote: post #64
dijit wrote: post #61

Earlier quoted context omitted.

Yes, I meant Jetstream (I even typed it but second guessed myself, my mistake) I’m typing these when I get a moment as I’m at a wedding- so I apologise. The issue in the docs was that there are no available Jetstream numbers, so I talked over a video call to the field CTO, who cited the base NATs numbers to me, and when I pressed him on if it was with Jetstream he said that it was without: so I asked for them with Je…

This doesn’t really support your position as far as most readers are concerned - it sounds like a disconnect. If they didn’t do this in any ad copy or public docs it’s not really in Mongo territory.

It’s deceptive if true, why are you trying to spin it as it’s ok cause the deception were not published

Re: Why was Apache Kafka created?

#72

Post by shikhar »

atombender wrote: post #70
varbhat wrote: post #18

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

NATS is very good. It's important to distinguish between core NATS and Jetstream, however. Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No dur…

> Jetstream clusters don't scale to many servers (they recommend max 3, I think)

Jetstream is even more limited than most Kafkas on number of streams https://github.com/nats-io/nats-server/discussions/5128#disc...

Re: Why was Apache Kafka created?

#73

Post by zbentley »

shikhar wrote: post #69
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

You might like what we are building with https://s2.dev :)

Looks interesting; does it take a different architectural approach than WarpStream did?

That’s not coded “you’re reinventing the wheel”; WarpStream had some significant drawbacks, so I’m truly curious about different approaches in the message-log-backed-by-blob-store space.

Re: Why was Apache Kafka created?

#74

Post by shikhar »

zbentley wrote: post #73
shikhar wrote: post #69

Earlier quoted context omitted.

You might like what we are building with https://s2.dev :)

Looks interesting; does it take a different architectural approach than WarpStream did? That’s not coded “you’re reinventing the wheel”; WarpStream had some significant drawbacks, so I’m truly curious about different approaches in the message-log-backed-by-blob-store space.

Architecturally, there are a lot of the same considerations. A key difference is that we offer streams as a cloud API, and it does not have WarpStream's BYOC split – where some stuff runs in your environment and metadata lives in their cloud – so we can offer lower latencies. We are also not trying to be Kafka API compatible, S2 has its own REST API.

The dimensions we focus on are number of streams (unlimited, so you can do granular streams like per user or session), internet accessibility (you can generate finely-scoped access tokens that can be safely used from clients like CLIs or browsers), and soon also massive read fanout for feed-like use cases.

Re: Why was Apache Kafka created?

#75

Post by throwaway8242 »

buckle8017 wrote: post #28
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Java the language and Java the runtime are fine. The way most Java code is written is terrible Enterprise factory factory factory.

That doesn't match my experience in the last 15 years working for 3 companies (one was a big enterprise, one medium sized and one startup)

Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.

Re: Why was Apache Kafka created?

#76

Post by slipperydippery »

jayd16 wrote: post #53
slipperydippery wrote: post #16

Earlier quoted context omitted.

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…

"Java is bloated because I only look at the bloated examples." Is C++ bloated because of the memory Chrome uses?

When all your examples in actual use are bloated…

I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C, but real programs are bloated and slow.

Re: Why was Apache Kafka created?

#77

Post by rvz »

AtlasBarfed wrote: post #54
rvz wrote: post #38

Earlier quoted context omitted.

> I'm curious, what exactly feels bloated about Java? Everything. Why do you think Kubernetes is NOT written in Java?

... Because it came from Google? Golang has little to distinguish itself technically. It has a more modern std lib (for now) and isn't Oracle. Which aren't trivial, but they aren't Trump cards.

> ... Because it came from Google?

Nope.

None of what you said are any of the reasons given that it WAS written in Java already [0] but rewrote it all in Go explicitly because of its performance, concurrency and single binary distribution characteristics.

Those were enough technical advantages to abandon any thought of a production-grade version of k8s in Java.

[0] https://archive.fosdem.org/2019/schedule/event/kubernetesclu...

Re: Why was Apache Kafka created?

#78

Post by betaby »

erulabs wrote: post #67

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was…

> Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great.

I sue Kafka precisely for that. Redis Cluster Streams have AOF persistence logs as I see from the doc. How stable it is?

Re: Why was Apache Kafka created?

#79

Post by EdwardDiego »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?"

The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.

Using S3 was better, but we still hit data loss problems (on our end, not S3 's, to clear) which isn't great when you need to bill on some of that data.

Then I heard about Kafka, bought a copy of I I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."

But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.

And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.

Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."

Re: Why was Apache Kafka created?

#80

Post by dxxvi »

erulabs wrote: post #67

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was…

> turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run

I don't have any experience with Redis Cluster Streams. Could you please tell us how it is simpler to manage? IMO, installing and managing a Kafka cluster in a non Fortune 100 scale is simple enough: run 1 java command for zookeeper, run another java command for a broker (with recent version of Kafka, zookeeper is not needed anymore). The configuration files are not very simple but not very complicated either. When we have another machine, we can run another broker on it.

Redis Cluster Streams is cheaper to run because it's written in C, doesn't need a VN to run? Or because its messages are stored in RAM not SSD?

Post reply on HN