Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

171–180 of 201 posts

Re: Kafka without ZooKeeper

#171

Post by chokolad »

travisjeffery wrote: post #139
didip wrote: post #73

Reminded me of this project: https://github.com/travisjeffery/jocko Kafka implemented in Go without needing Zookeeper.

Hey, I'm the author of Jocko. I've been working at Confluent the past four years. I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-g...

Heh, I just bought it based on PragProg newsletter :). Literally 5 minutes ago.

Re: Kafka without ZooKeeper

#172

Post by thibauts »

agallego wrote: post #162
thibauts wrote: post #145

Earlier quoted context omitted.

I’m trying to take a similar path with this project [1] though there’s no HA story for now and it’s not meant to be protocol-compatible. The goal is to make it lightweight and simple to operate, yet very fast. [1] https://github.com/dataptive/styx

From a feature list perspective on the readme it basically is what we did at Redpanda (dev here)

Hi Alexander ! I'm a big fan of the work you do at vectorizedio and of your blog posts ! I strongly believe there's a need for much simpler event streaming and room for improvement performance-wise.

I took a different path by divorcing from the Kafka protocol and experimenting with what I believe is a simpler to model approach to reliable event processing [1].

[1] https://github.com/dataptive/styx/blob/master/docs/howto/rel...

Re: Kafka without ZooKeeper

#173

Post by gigatexal »

mdaniel wrote: post #169
sgt wrote: post #131

Earlier quoted context omitted.

What's the downside?

The BSL license makes choosing it "a discussion" versus using OSI licensed software: https://github.com/vectorizedio/redpanda/tree/v21.3.7/licens... I don't have enough experience with these new vanity licenses to know what the contribution story looks like, either

What about the BSL makes it a deal breaker? You can host it yourself you just can’t compete with their hosted offering to other customers. I think that’s fair: they get to monetize what they make. And then after 3 years iirc all the code that was BSL or that version anyway goes even more permissive.

Re: Kafka without ZooKeeper

#174

Post by skyde »

closeparen wrote: post #95
airhead969 wrote: post #3

IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.

It seems weird to me that cloud providers do not offer distributed coordination primitives "as a service." I understand there are KV stores but not with watches, locks, etc. in the way that etcd and ZK have them.

Are you looking for something like this ? https://azure.microsoft.com/en-us/updates/azure-cosmos-db-ap...

Re: Kafka without ZooKeeper

#175

Post by skyde »

tveita wrote: post #80
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. (That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)

I think you are completely missing the point. Those kind of system need to be designed in layer. You can host all 3 layer on each of your VM instance if you want but they should not be mixed in the same process.

One layer BookKeeper provide an abstraction similar to HDFS. That is it provide file that are horizontally scalable in size and throughput and reliable append only files.

Pulsar is a service built on top of BookKeeper but could run on top of HDFS or something like Amazon S3 ...

And is only responsible for making sure there is only one writer per BookKeeper file even if multiple process try sending request to Pulsar to write to the same partition. It also try to balance request across all the brokers.

Re: Kafka without ZooKeeper

#176

Post by jganetsk »

dividedbyzero wrote: post #85
alex_anglin wrote: post #60

Earlier quoted context omitted.

Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?

Is there a GCP offering that isn't just Confluent Cloud billed via Google?

You can use Pub/Sub Lite: https://cloud.google.com/pubsub/lite/docs

With a Kafka compatibility shim: https://github.com/googleapis/java-pubsublite-kafka

Disclaimer: I work on GCP.

Re: Kafka without ZooKeeper

#177

Post by jganetsk »

haolez wrote: post #42
cmason wrote: post #5

Earlier quoted context omitted.

What do you use instead?

Cheap managed cloud services, like AWS SQS and Azure Storage Queue (I usually want some kind of persistence for my queues).

Pub/Sub Lite is a cheap managed cloud service on GCP: https://cloud.google.com/pubsub/lite/docs

With a Kafka compatibility shim: https://github.com/googleapis/java-pubsublite-kafka

Disclaimer: I work for GCP.

Re: Kafka without ZooKeeper

#178

Post by menzella-g »

cmason wrote: post #5
haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

What do you use instead?

GCP offers Pub/Sub Lite, an inexpensive messaging product with Kafka-compatible client libraries.

https://cloud.google.com/pubsub/lite/docs https://github.com/googleapis/java-pubsublite-kafka

Disclaimer: I work on this product.

Re: Kafka without ZooKeeper

#179

Post by closeparen »

skyde wrote: post #174
closeparen wrote: post #95

Earlier quoted context omitted.

It seems weird to me that cloud providers do not offer distributed coordination primitives "as a service." I understand there are KV stores but not with watches, locks, etc. in the way that etcd and ZK have them.

Are you looking for something like this ? https://azure.microsoft.com/en-us/updates/azure-cosmos-db-ap...

Yes! Exactly. Are there analogues in GCP and AWS?
Post reply on HN