Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

11–20 of 295 posts

Re: Kafka is dead, long live Kafka

#11

Post by ryanworl »

cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

[WarpStream co-founder and CTO here]

1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our cloud control plane.

2. We think people shouldn't have to choose between reliability and costs. WarpStream gives you the reliability and availability of running in three AZs but with the cost of one.

3. We have a custom metadata database running in our cloud control plane which handles ordering.

Re: Kafka is dead, long live Kafka

#12

Post by richieartoul »

morelisp wrote: post #10
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

I'm also especially interested in 3) - from the arch overview it sounds like all agents are actively writing and actively compacting, how do they coordinate which topic-partitions to compact? Is the Cloud Metadata Store essentially responsible for handing out the offsets?

(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc

Re: Kafka is dead, long live Kafka

#13

Post by winddude »

ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

Kafka itself no longer requires zookeeper.

Re: Kafka is dead, long live Kafka

#14

Post by slantedview »

This article seems to be spreading some FUD, in particular this comment:

> Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs.

You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.

Re: Kafka is dead, long live Kafka

#15

Post by richieartoul »

dottedmag wrote: post #5
ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

Do you have a reference documentation for S3 data layout?

(WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.

Re: Kafka is dead, long live Kafka

#16

Post by rcarmo »

Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive.

I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant, AZ-redundant or globally redundant storage, most of which are available to run managed disks upon).

This is why properly designed Hadoop/Kafka cloud managed services employ storage adapters to leverage the provider's baked in redundancy. And why some cloud providers have Kafka-compatible event brokers.

The rest of what WarpStream does is just icing on the cake (although I'm curious as to the internals and how they avoid inter-AZ charges).

(full disclosure: I work at Microsoft, but built Hadoop/Spark/Kafka clusters before joining nearly a decade ago.)

Re: Kafka is dead, long live Kafka

#17

Post by vasco »

ilikehurdles wrote: post #9
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

Re #1 I don’t think Amazon charges for data transfers from within AWS, assuming Kafka is hosted there.

S3 requests are paid regardless of source (tldr GETs are cheaper than PUTs). You're confused with data transfer within a VPC.

Re: Kafka is dead, long live Kafka

#18

Post by finnh »

ilikehurdles wrote: post #9
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

Re #1 I don’t think Amazon charges for data transfers from within AWS, assuming Kafka is hosted there.

There are per-API-call charges. It's just not per-byte.

Re: Kafka is dead, long live Kafka

#19

Post by richieartoul »

slantedview wrote: post #14

This article seems to be spreading some FUD, in particular this comment: > Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.

(WarpStream Founder) That's fair, but we think people shouldn't have to choose between multiple AZs and costs. All cloud providers recommend that critical workloads running in cloud environments should run in more than one AZ.

Re: Kafka is dead, long live Kafka

#20

Post by ant6n »

I keep seeing these HN titles about technologies named for other concepts, thinking it's talking about those other contexts, without any indication it's about those obscure technologies. Then I'm disappointed its just about some random, badly named technology.

So, in honor of actual Zombie Kafka, here a short news blurb about Kafka international airport being Named most Alienating Airport: https://www.youtube.com/watch?v=gEyFH-a-XoQ

Post reply on HN