Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

81–90 of 295 posts

Re: Kafka is dead, long live Kafka

#81

Post by derefr »

rubiquity wrote: post #3
ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

Congrats! "The SQLite of Kafka" is an item from my side projects pile I'm happy to delete. One reason I never built it is because it felt paradoxical that users might want a scaled down Kafka rather than using SQLite directly if the scale didn't matter. But you may find out that people enjoy the semantics of the Kafka protocol or are already using Kafka and have learned they don't have the scale they thought they did…

> it felt paradoxical that users might want a scaled down Kafka rather than using SQLite directly if the scale didn't matter.

I don't need to push very many messages (not enough to justify running Kafka), but each of the messages that I do push are both 1. very important and must be cross-AZ durable, and 2. very urgent and must not be blocked by e.g. contended writes in a regular RDBMS.

Currently, the winner of this use-case for IaaS customers is "whatever cloud-native message-queue service your IaaS offers." (And those customers would also be the extent of WarpStream's Total Addressable Market here, given that WarpStream's architecture fundamentally relies on having a highly-replicated managed object store available.)

I'm therefore curious: in what ways does WarpStream win vs. Amazon SQS / Google Cloud Pub/Sub / Azure Queue Storage?

Re: Kafka is dead, long live Kafka

#82

Post by derefr »

nithril wrote: post #69
rcarmo wrote: post #16

Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant,…

> Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? Azure disk replications are for the durability of the data, not the avalability of the data from a kafka perspective.

Many systems might just be using Kafka to drive async batch transaction processing (think: sending emails, charging credit cards), and therefore don't care at all about availability.

Re: Kafka is dead, long live Kafka

#83

Post by richieartoul »

derefr wrote: post #76
ryanworl wrote: post #33

Earlier quoted context omitted.

[WarpStream co-founder here] That is correct about flushing. RE: consuming. The TLDR; is that the agents in an availability zone cluster with each other to form a distributed file cache such that no matter how many consumers you attach to a topic, you will almost never pay for more than 1 GET request per 4MiB of data, per zone. Basically when a consumer fetches a block of data for a single partition, that will trigge…

Is there a reason you built that cache layer yourself (rather than each node "just" running its own sidecar MinIO instance, that write-throughs to the origin object store?)

(WarpStream co-founder)

The cache is for reads, not writes. There is no cache for writes.

We built our own because it needed to behave in a very specific way to meet our cost/latency goals. Running a MinIO sidecar instance means that every agent would effectively have to download every file in its entirety which would not scale well and would be expensive. We also have a pretty hard and fast rule about keeping deploying WarpStream as simple as rolling out a single stateless binary.

Re: Kafka is dead, long live Kafka

#85

Post by sgt »

richieartoul wrote: post #73
sgt wrote: post #72

Can this be used without S3?

(WarpStream co-founder) It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the agent. For local development you can use an in-memory or local file storage, but not for "real" usage.

Okay - yes, it would make a lot of sense as a Kafka alternative to those of us who prefer to stay away from the public clouds.

Re: Kafka is dead, long live Kafka

#87

Post by derefr »

As someone who's been doing async-fanout physical-replication of Postgres instances for the longest time using pgBackRest's S3 repo support (the primary writes to S3; the replicas read from S3; the primary's uplink doesn't get saturated serving the writes) I've always wondered where the equivalent for message-queue systems was. Glad to see it :)

Re: Kafka is dead, long live Kafka

#88

Post by parhamn »

slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

Kafka is a closer to a persistent WAL than a message queue. If your work doesn't need a WAL, it's almost certainly overkill and you will hate it. If your work needs a WAL then it'll be your favorite tool ever.

Re: Kafka is dead, long live Kafka

#89

Post by jolux »

Haven’t seen it mentioned yet so I’ll just say we use Kinesis Data Streams with enhanced fanout at work and it seems very minimal in terms of operational overhead if it fits your needs.

Re: Kafka is dead, long live Kafka

#90

Post by pdeva1 »

richieartoul wrote: post #35
pdeva1 wrote: post #32

Earlier quoted context omitted.

1. dont producers now have much higher latency since they have to wait for writes to s3. 2. if the '5-10x cheaper' is mostly due to cross AZ savings, isnt that offered by AWS MSK offering too?

(WarpStream founder) 1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also, you still have to more or less manage a Kafka cluster (hot spotting, partition rebalancing, etc). In practice we think WarpStream will be much cheaper to use than MSK for almost…

1. what payload size and flush interval is that latency measured against?
Post reply on HN