Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

31–40 of 295 posts

Re: Kafka is dead, long live Kafka

#31

Post by tekla »

ant6n wrote: post #20

I keep seeing these HN titles about technologies named for other concepts, thinking it's talking about those other contexts, without any indication it's about those obscure technologies. Then I'm disappointed its just about some random, badly named technology. So, in honor of actual Zombie Kafka, here a short news blurb about Kafka international airport being Named most Alienating Airport: https://www.youtube.com/wat…

Seriously. I ordered Windows and all I got was a piece of cardboard with some random letters and numbers on it, and not a frame with glass.

Re: Kafka is dead, long live Kafka

#32

Post by pdeva1 »

ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

1. dont producers now have much higher latency since they have to wait for writes to s3.

2. if the '5-10x cheaper' is mostly due to cross AZ savings, isnt that offered by AWS MSK offering too?

Re: Kafka is dead, long live Kafka

#33

Post by ryanworl »

theikkila wrote: post #30
ryanworl wrote: post #11

Earlier quoted context omitted.

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…

Related to 1. If I understood corrently the agent generates single object per each flushing interval containing all data accross all topics it has received. Does this mean that when reading the consumer needs to read multiple partition data simultaneously to access just single partition? How about scaling consumers horizontally how does WarpStream Agent handle horizontal partitioning of the stream from consuming side…

[WarpStream co-founder here]

That is correct about flushing. RE: consuming. The TLDR; is that the agents in an availability zone cluster with each other to form a distributed file cache such that no matter how many consumers you attach to a topic, you will almost never pay for more than 1 GET request per 4MiB of data, per zone. Basically when a consumer fetches a block of data for a single partition, that will trigger an "over read" of up to 4MiB of data that is then cached for subsequent requests. This cache is "smart" and will deduplicate all concurrent requests for the same 4MiB blocks across all agents within an AZ.

It's a bit difficult to explain succinctly in an HN comment, but hopefully that helps.

Re: Kafka is dead, long live Kafka

#34

Post by dmytton »

The blog post mentions "P99 of ~1s of producer-to-consumer latency". What about just producer latency i.e. message successfully received into the queue ready to be picked up? S3 writes seem be in the low 100s of ms so I assume that's part of the quoted end-to-end latency.

Re: Kafka is dead, long live Kafka

#35

Post by richieartoul »

pdeva1 wrote: post #32
ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

1. dont producers now have much higher latency since they have to wait for writes to s3. 2. if the '5-10x cheaper' is mostly due to cross AZ savings, isnt that offered by AWS MSK offering too?

(WarpStream founder)

1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also, you still have to more or less manage a Kafka cluster (hot spotting, partition rebalancing, etc). In practice we think WarpStream will be much cheaper to use than MSK for almost all use-cases, and significantly easier to manage.

Re: Kafka is dead, long live Kafka

#36

Post by richieartoul »

dmytton wrote: post #34

The blog post mentions "P99 of ~1s of producer-to-consumer latency". What about just producer latency i.e. message successfully received into the queue ready to be picked up? S3 writes seem be in the low 100s of ms so I assume that's part of the quoted end-to-end latency.

(WarpStream co-founder)

Yeah our P99 for producer latency is ~400ms right now.

Re: Kafka is dead, long live Kafka

#37

Post by cannonpalms »

aduffy wrote: post #7
episiarch wrote: post #4

Earlier quoted context omitted.

How do you replace ZooKeeper?

Kafka replaced ZooKeeper with Kafka itself already a few years ago https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

And first announced to be "production ready" in October 2022.

Re: Kafka is dead, long live Kafka

#39

Post by pshirshov »

Ok, intsead of providing commit log guarantees on their own they rely on S3.

I'm not sure if S3 can actually provide durable commit log guarantees.

There is timestamp-based last write wins policy for concurrent writes, so I'm not sure if this thingy can actually replace Kafka in all the usecases providing same guarantees.

Re: Kafka is dead, long live Kafka

#40

Post by theikkila »

ryanworl wrote: post #11
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…

Does WarpStream guarantee correct order inside partition only for acknowledged messages or also among the acknowledged messages (in different batches)? If so how do you keep clocks synchronized between the agents?
Post reply on HN