It'd be great if there was an rss feed for your blog
Kafka is dead, long live Kafka
21–30 of 295 posts
Re: Kafka is dead, long live Kafka
#22I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.
How do you replace ZooKeeper?
(WarpStream founder) WarpStream has a completely different architecture than Kafka: https://docs.warpstream.com/warpstream/background-informatio...
That said it does require a lot of metadata to orchestrate all the different concurrent operations over S3. We handle this with a custom metadata store that we run in our cloud control plane.
Re: Kafka is dead, long live Kafka
#23I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.
The blog post mentions that partitions are too low-level an abstraction to program against. Does that mean WarpStream doesn't use partitions? Do you provide any ordering guarantees like Kafka does at the partition level?
(WarpStream founder) No WarpStream has partitions internally and provides the same ordering guarantees Kafka does at the partition level. We're just saying that we think for most streaming applications this is not a great programming model, and we think there is an opportunity to do something better (but we haven't done that yet).
Re: Kafka is dead, long live Kafka
#24Earlier quoted context omitted.
I'm also especially interested in 3) - from the arch overview it sounds like all agents are actively writing and actively compacting, how do they coordinate which topic-partitions to compact? Is the Cloud Metadata Store essentially responsible for handing out the offsets?
(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc
I see - this also explains the tiny consumer group / topic-partition limit of the free plan, then...
Re: Kafka is dead, long live Kafka
#25Earlier quoted context omitted.
Re #1 I don’t think Amazon charges for data transfers from within AWS, assuming Kafka is hosted there.
There are per-API-call charges. It's just not per-byte.
(WarpStream founder) Exactly, and we designed WarpStream's storage system to minimize the number of object storage API calls it has to make while still maintaining relatively low latency (P99 of ~1s end-to-end)
Re: Kafka is dead, long live Kafka
#26Is the plan to eventually open source this and provide paid support?
Re: Kafka is dead, long live Kafka
#27company competing with kafka says kafka is dead, is that accurate?
Re: Kafka is dead, long live Kafka
#28Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.
Re: Kafka is dead, long live Kafka
#29Earlier quoted context omitted.
(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc
I see - this also explains the tiny consumer group / topic-partition limit of the free plan, then...
[WarpStream co-founder here]
What do you think would be a good limit for the free plan?
This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitrary number to start with.
Re: Kafka is dead, long live Kafka
#30Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…
[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…
Related to 1. If I understood corrently the agent generates single object per each flushing interval containing all data accross all topics it has received. Does this mean that when reading the consumer needs to read multiple partition data simultaneously to access just single partition? How about scaling consumers horizontally how does WarpStream Agent handle horizontal partitioning of the stream from consuming side?