Works very well. Had some time to try out the 2.8 build and all existing tooling works just fine. Solid work by everyone involved in this.
Kafka without ZooKeeper
191–200 of 201 posts
Re: Kafka without ZooKeeper
#192Reminded me of this project: https://github.com/travisjeffery/jocko Kafka implemented in Go without needing Zookeeper.
Hey, I'm the author of Jocko. I've been working at Confluent the past four years. I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-g...
Sounds just like the kind of book I wanna read. Unfortunately I wasn't able to pay for it due to "merchant configuration".
I imagine there are no region restrictions, right? I've already contacted support though.
Re: Kafka without ZooKeeper
#193Cool feature! I have a concern, if the Kafka broker provide both coordination service and Kafka service, how to achieve the resource isolation? If some of the topic which on the coordination service with very high throughput, this must cause instability of the coordination service, could this further cause the instability of the entire cluster? If some of the broker only provide the coordination service, what is this…
Part of the new design is that brokers and controllers can be run on separate JVMs.
Re: Kafka without ZooKeeper
#194Earlier quoted context omitted.
> You end up splitting your data across partitions anyways Messages are ordered within partitions. > What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle. TCP/IP ?
> Messages are ordered within partitions. Right, but that means you're still "unordered" across those partitions? > TCP/IP ? But TCP/IP isn't delivered in order, it rearranges the unordered packages by their ID. I guess ordered delivery would be nice for that, but I just feel like making your protocol not require ordering is far simpler. Not to mention that both TCP and Kafka have to handle head of line blocking? I'm…
> Right, but that means you're still "unordered" across those partitions?
Right, so related messages have an ordering guarantee but unrelated messages may be processed out of order relative to each other, which is usually what you want. (Of course you do have to set the record key correctly).
> I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons.
It's a lot more lightweight than full ACID, but if you get your dataflow right it achieves everything that a traditional database does. Without ordering you wouldn't be able to do anything that requires any kind of consistency.
Re: Kafka without ZooKeeper
#195Cool feature! I have a concern, if the Kafka broker provide both coordination service and Kafka service, how to achieve the resource isolation? If some of the topic which on the coordination service with very high throughput, this must cause instability of the coordination service, could this further cause the instability of the entire cluster? If some of the broker only provide the coordination service, what is this…
Part of the new design is that brokers and controllers can be run on separate JVMs.
Yes, that my second concern. If run on separate JVM, what is the difference between use ZK and use controller? Of course, If the new controller is more stable than ZK, more efficient than ZK, this will bring benefits to users. But this is essentially replacing zk with a better product, I think etcd also can achieve this purpose.
Re: Kafka without ZooKeeper
#196Earlier quoted context omitted.
I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client suppor…
> The broader Kafka ecosystem has features like exactly-once support No. No it doesn’t. It has at-least-once delivery with client-side deduplication. That’s not new, it’s what TCP does FFS. Why would you lie to people about supporting something long established at best and demonstrably impossible at worst? > Finally, I work at Confluent.... Oh, that’s why. Never mind then. Continue selling digital snake oil.
> No. No it doesn’t. It has at-least-once delivery with client-side deduplication. That’s not new, it’s what TCP does FFS.
It's not just deduplication, you can atomically commit a consumer from one topic + produce of records resulting from that. Which is exactly the same exactly-once guarantee that you get from e.g. an SQL database in linearizable mode (a lot of SQL databases will do the same thing internally - optimistically execute transactions and then re-run them in the case of a conflict).
Re: Kafka without ZooKeeper
#197Earlier quoted context omitted.
Hey, I'm the author of Jocko. I've been working at Confluent the past four years. I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-g...
Sounds just like the kind of book I wanna read. Unfortunately I wasn't able to pay for it due to "merchant configuration". I imagine there are no region restrictions, right? I've already contacted support though.
I'm not sure, I haven't heard of that happening before. I've asked my editor if she knows what's up. You can email me at tj at my HN username dot com if you want to follow up.
Re: Kafka without ZooKeeper
#198Earlier quoted context omitted.
The BSL license makes choosing it "a discussion" versus using OSI licensed software: https://github.com/vectorizedio/redpanda/tree/v21.3.7/licens... I don't have enough experience with these new vanity licenses to know what the contribution story looks like, either
What about the BSL makes it a deal breaker? You can host it yourself you just can’t compete with their hosted offering to other customers. I think that’s fair: they get to monetize what they make. And then after 3 years iirc all the code that was BSL or that version anyway goes even more permissive.
I avoided language such as "deal breaker" or "problematic" and chose rather to just say "it would require discussion" in ways that I have not once ever encountered with an Apache licensed project.
Re: Kafka without ZooKeeper
#199Earlier quoted context omitted.
> Messages are ordered within partitions. Right, but that means you're still "unordered" across those partitions? > TCP/IP ? But TCP/IP isn't delivered in order, it rearranges the unordered packages by their ID. I guess ordered delivery would be nice for that, but I just feel like making your protocol not require ordering is far simpler. Not to mention that both TCP and Kafka have to handle head of line blocking? I'm…
> Right, but that means you're still "unordered" across those partitions? Right, so related messages have an ordering guarantee but unrelated messages may be processed out of order relative to each other, which is usually what you want. (Of course you do have to set the record key correctly). > I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons. It's a lot more lightweigh…
Hm, ok, yeah. So I guess I can see what you mean. I've never had a use case where I felt comfortable relying on any kind of message ordering, and always rely on my application level logic to handle that, or ensure the system is resilient despite ordering (ie: commutative operations only).
To me, it seemed at odds with the parallelism of a partition, but I suppose in this case you'd be partitioning on some sort of semantic key vs, say, a hash.
Thanks for bearing with me on that, this was just an unfamiliar idea for me.
Re: Kafka without ZooKeeper
#200Earlier quoted context omitted.
Sounds just like the kind of book I wanna read. Unfortunately I wasn't able to pay for it due to "merchant configuration". I imagine there are no region restrictions, right? I've already contacted support though.
I'm not sure, I haven't heard of that happening before. I've asked my editor if she knows what's up. You can email me at tj at my HN username dot com if you want to follow up.
I was able to get it sorted through support. Thank you :)