Question : does this architecture have any difference regarding ordering or durability of messages, compared to kafka ?
Kafka is dead, long live Kafka
51–60 of 295 posts
Re: Kafka is dead, long live Kafka
#52Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.
What? It’s pretty easy to set up and get going
Re: Kafka is dead, long live Kafka
#53Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…
[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…
[dead]
Re: Kafka is dead, long live Kafka
#54company competing with kafka says kafka is dead, is that accurate?
No
Re: Kafka is dead, long live Kafka
#55[deleted]
Re: Kafka is dead, long live Kafka
#56I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 ( https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c... ) but this approach is so much better. Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible. Very cool, have to find some time to take this for a test drive.
(WarpStream co-founder)
Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!
Re: Kafka is dead, long live Kafka
#57Have you considered making a plain HTTPS frontend like Confluent's REST proxy? Or even just designating a bucket as 'incoming messages' and letting people use the S3 API to write messages?
Kafka is great but if you're targeting Amazon customers, you're competing with SQS too.
Re: Kafka is dead, long live Kafka
#58Question : does this architecture have any difference regarding ordering or durability of messages, compared to kafka ?
(WarpStream co-founder)
Nope, we provide the same durability and ordering messages as Kafka, albeit with higher latency. We never acknowledge produce requests until they've been durably persisted in object storage (P99 ~400ms), so its the same guarantee as running a 3AZ Kafka cluster with fsync enabled.
Re: Kafka is dead, long live Kafka
#59Pretty cool. I'm definitely going to check this out.
Re: Kafka is dead, long live Kafka
#60It's refreshing to see cloud computing costs matter. Hopefully it'll cut down of needless cloud complexity and the proliferation of indirection in software abstractions of the last decade.