Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

261–270 of 295 posts

Re: Kafka is dead, long live Kafka

#261

Post by ryanworl »

chaotic-good wrote: post #260
ryanworl wrote: post #124

Earlier quoted context omitted.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

It could be easy to operate when everything is fine but what's about incidents? If I understand correctly, there is a metadata database (BTW, is it multi-AZ as well?). But what if there is a data loss incident and some metadata was lost? Is it possible to recover from S3? If this is possible, then I guess that can't be very simple and should require a lot of time because S3 is not that easy to scan to find all the ar…

[WarpStream CTO here]

I'm not going to respond to your comment directly (we've already solved all the problems you've mentioned), but I thought I should mention for the sake of the other readers of this thread that you work for Redpanda which is a competitor of ours and didn't disclose that fact. Not a great look.

https://github.com/Lazin

Re: Kafka is dead, long live Kafka

#262

Post by chaotic-good »

rcarmo wrote: post #16

Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant,…

> Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system?

That's easy. EBS and similar solutions comes with the price. They're very expensive. Especially, when you need a lot of IOPs. You may be saving on cross-AZ traffic but you will pay ridiculous amount of money on storage. If you have replication you can use attached storage which is way cheaper.

Re: Kafka is dead, long live Kafka

#263

Post by mdaniel »

physicles wrote: post #240
EdwardDiego wrote: post #153

Earlier quoted context omitted.

ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…

Not my experience at all. We’ve been running a 3 node cluster for several years, and a significant minority of the times I’ve been paged is because ZK got into a bad state that was fixed by a restart (what bad state exactly? Don’t know, don’t care, don’t have two spare weeks to spend figuring it out). Note that we have proper liveness checks on individual instances, so the issue is more complicated than that. Migrate…

> Migrated to 3.3 with KRaft about half a year ago

Did you follow their migration guide, or did you just rebuild the cluster and then using KRaft? I didn't know how "migration" was used in that context

Re: Kafka is dead, long live Kafka

#264

Post by huac »

ryanworl wrote: post #124
poorman wrote: post #121

Earlier quoted context omitted.

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

Is it possible to have a 'knob' here? some topics might need low latency even if most don't. My sense, reading this, is that while most topics / use cases will be fine on Warpstream, that some will not be.

Re: Kafka is dead, long live Kafka

#265

Post by red-iron-pine »

EdwardDiego wrote: post #152
snapetom wrote: post #106

Earlier quoted context omitted.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

Kafka isn't even a queue. I've done consulting on Kafka, and several times my recommendation is "You don't want or need Kafka". A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

"but MQ is, like, so 1998, we want new and cool and GCP told us we needed Kafka"

Re: Kafka is dead, long live Kafka

#266

Post by mdaniel »

purpleblue wrote: post #103

What's to stop Confluent/Kafka from simply implementing a feature like this themselves?

their recent blog post doesn't sound that far off from the OP's architecture: https://www.confluent.io/blog/cloud-native-data-streaming-ka... (discussed: https://news.ycombinator.com/item?id=36870278 )

Re: Kafka is dead, long live Kafka

#267

Post by aftermodern »

The object storage backend is interesting and I’m wondering if Kafka will have a native option for that, eventually.

Confluent hasn’t outright advertised it, but one of their engineers mentioned in a q&a that their Cloud offering is using object storage for their own backend now.

Re: Kafka is dead, long live Kafka

#268

Post by dtjohnnymonkey »

I've seen both your FoundationDB talks, curious if you are using FoundationDB under the hood or how much of the metadata store is homegrown?

Also, how did you implement the parsing of the Kafka protocol? From scratch or pieced together with open source Go bits? Having a nice programmable API for building Kafka-compatible servers could be a huge boon to the Kafka community (wink wink).

Re: Kafka is dead, long live Kafka

#269

Post by chaotic-good »

ryanworl wrote: post #261
chaotic-good wrote: post #260

Earlier quoted context omitted.

It could be easy to operate when everything is fine but what's about incidents? If I understand correctly, there is a metadata database (BTW, is it multi-AZ as well?). But what if there is a data loss incident and some metadata was lost? Is it possible to recover from S3? If this is possible, then I guess that can't be very simple and should require a lot of time because S3 is not that easy to scan to find all the ar…

[WarpStream CTO here] I'm not going to respond to your comment directly (we've already solved all the problems you've mentioned), but I thought I should mention for the sake of the other readers of this thread that you work for Redpanda which is a competitor of ours and didn't disclose that fact. Not a great look. https://github.com/Lazin

I'm not asking anything on behalf of any company and just genuinely curious (and I don't think that we're competitors, both systems are designed for totally different niches). I'm working on tiered-storage implementation btw. Looks like the approach here is the total opposite of what everyone else is doing. I see some advantages but also disadvantages to this. Hence the question.

Re: Kafka is dead, long live Kafka

#270

Post by relay23 »

aftermodern wrote: post #267

The object storage backend is interesting and I’m wondering if Kafka will have a native option for that, eventually. Confluent hasn’t outright advertised it, but one of their engineers mentioned in a q&a that their Cloud offering is using object storage for their own backend now.

They always have.. you get it but you don't know it.
Post reply on HN