211–220 of 295 posts
This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit . FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could impl…
Earlier quoted context omitted.
> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not https://kafka.apache.org/documentation/#zk > makes keeping two distributed systems alive and healthy, but don't worry you can't move your production cluster…
ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…
That has not been my experience. I've been running several small cluster (3 and 5 node) Confluent packaged for the last 3 years, and zookeeper ~20 times has gotten into this state where a node isn't in the cluster, and the way to "fix" it is to restart the current leader node. Usually I have to play "whack-a-mole" until I've restarted enough leaders that it comes up. Sometimes I've not been able to get the node back into the cluster without shutting down the whole cluster and restarting it.
Once it's running it's fine, until updates are done. But this getting into a weird state sure doesn't sit well with me.
We build an admin console / dev tooling for Kafka (https://kpow.io) that supports Kafka 1.0+ including Redpanda due to their fairly strict adherence to those API.
Warpstream seems like a cool idea, I'd like to see what happens if we plug Kpow on top of it, if that's possible.
Hi @richieartoul, does warpstream support the standard Kafka Admin API? We build an admin console / dev tooling for Kafka ( https://kpow.io ) that supports Kafka 1.0+ including Redpanda due to their fairly strict adherence to those API. Warpstream seems like a cool idea, I'd like to see what happens if we plug Kpow on top of it, if that's possible.
That said we’d be stoked to get this working as an additional tool for people. Do you want to shoot me an email or join our slack so we can discuss further? I can probably prioritize whatever protocol features were missing to get it working.
founders@warpstreamlabs.com
We've built something probably very similar to this product at my previous job. We had double digit TB daily ML traffic that didn't require realtime latency, so we moved those all onto S3 and also saw approx ~90% cost savings. This was built on top of JVM and still used a 6 broker Kafka cluster to keep metadata (vs probably 300 when it was originally all on Kafka). Kafka's compute and storage model doesn't scale too…
Yeah we’ve run into a number of people who’ve rolled their own solution in this space. The “push pointers to S3 through traditional Kafka” approach is a very practical one.
Was this memq at Pinterest, or something else?
Earlier quoted context omitted.
> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not https://kafka.apache.org/documentation/#zk > makes keeping two distributed systems alive and healthy, but don't worry you can't move your production cluster…
ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…
I’m agnostic about Kafka but ZooKeeper is problematic for many use cases based on personal experience and I wouldn’t recommend it. It can be “rock solid” and still not very good. I’ve seen ZK replaced with alternatives at a few different organizations now because it didn’t work well in practice, and what it was replaced with worked much better in every case.
ZooKeeper works, sort of, but I wouldn’t call it “good” in some objective sense.
Earlier quoted context omitted.
Does WarpStream guarantee correct order inside partition only for acknowledged messages or also among the acknowledged messages (in different batches)? If so how do you keep clocks synchronized between the agents?
(WarpStream founder) It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps: 1. Write a file to S3 2. "Commit" that file to the meta…
We've built something probably very similar to this product at my previous job. We had double digit TB daily ML traffic that didn't require realtime latency, so we moved those all onto S3 and also saw approx ~90% cost savings. This was built on top of JVM and still used a 6 broker Kafka cluster to keep metadata (vs probably 300 when it was originally all on Kafka). Kafka's compute and storage model doesn't scale too…
(WarpStream co-founder) Yeah we’ve run into a number of people who’ve rolled their own solution in this space. The “push pointers to S3 through traditional Kafka” approach is a very practical one. Was this memq at Pinterest, or something else?