41–50 of 295 posts
This article seems to be spreading some FUD, in particular this comment: > Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.
Earlier quoted context omitted.
[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…
Does WarpStream guarantee correct order inside partition only for acknowledged messages or also among the acknowledged messages (in different batches)? If so how do you keep clocks synchronized between the agents?
It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps:
1. Write a file to S3 2. "Commit" that file to the metadata store which will then assign the partitions at commit time 3. Return the assigned partitions to the client
The title of the article should be "Kafka is dead. Long live Warpstream." The long live part refers to the successor.
We’ve had bare metal for a very long time now and it seems that managing your own bare metal hasn’t become much easier. If it were very easy, we’d see more of these sort of things be managed by the end user. That being said how are you managing this service? A cloud provider or bare metal?
Both you and Ryan have much experience with foundationdb which generally is managed yourself. Speaking of which, did you go with that for your metadata store again? Why or why not?
Ok, intsead of providing commit log guarantees on their own they rely on S3. I'm not sure if S3 can actually provide durable commit log guarantees. There is timestamp-based last write wins policy for concurrent writes, so I'm not sure if this thingy can actually replace Kafka in all the usecases providing same guarantees.
We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than once or overwrite an existing file.
Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant,…
So the table stakes for running a cluster was not 3x as much hardware but closer to 2.2x, which is a huge deal for solutions in the small and developer sandboxes. It also matters when 3 shards don't quite cover your load but 5 is too many. Or 6 vs 7.
The problem is that with geographic replication, this doesn't fix either of the problems articulated as part of the thesis of this article:
1. Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs.
2. Operational overhead – running your own Kafka cluster literally requires a dedicated team and sophisticated custom tooling.
Still, we need this functionality back for cloud, particularly as the pendulum swings back to self hosting, which it always has in the past.
This looks great but reading the accidental SRE gave me two questions: We’ve had bare metal for a very long time now and it seems that managing your own bare metal hasn’t become much easier. If it were very easy, we’d see more of these sort of things be managed by the end user. That being said how are you managing this service? A cloud provider or bare metal? Both you and Ryan have much experience with foundationdb w…
WarpStream's current offering is a hybrid BYOC approach. The customer runs the agents in their cloud account, and we manage the metadata store for them remotely. This keeps all the customer data in their cloud account and their S3 bucket where we can't see it or touch it. It does mean that the customer has to run the WarpStream agents themselves, but they're just stateless containers that are pretty easy to manage.
We considered using FoundationDB for our metadata store, but ended up not in the end. In order to make our free tier cost effective we really need to make our metadata store as efficient as possible for this specific use-case which required something a bit more custom. That said FoundationDB is a fantastic piece of technology. Best distributed database I've ever used, and I've used many :)
Earlier quoted context omitted.
Do you have a reference documentation for S3 data layout?
(WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.
1. Any plans on open-sourcing this?
2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases.
3. Related to 2, how does WarpStream compare to tiered storage in Pulsar?
[Edit 1] Added (3)
Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible.
Very cool, have to find some time to take this for a test drive.