Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

181–190 of 201 posts

Re: Kafka without ZooKeeper

#181

Post by skyde »

EdwardDiego wrote: post #110
nick0garvey wrote: post #109

Earlier quoted context omitted.

Some workloads are very CPU intensive and some are not. Being forced to scale CPU & disk together means one of them is going to be overprovisioned - often by a lot. I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.

Okay, I can see that point, but is it worth the additional latency between broker and Bookie? > I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases. Yeah, I think they were too tbh. I wish I could delve more into what they experienced beyond that single blog post I linked.

Latency is actually reduced compared to kafka because when you have 3 replica you just need to wait for a response of the fastest of the 3 replica instead of waiting for a response of the slowest (master) of the 3 replica.

Re: Kafka without ZooKeeper

#182

Post by [deleted] »

pc86 wrote: post #155
cmckn wrote: post #64

Earlier quoted context omitted.

Hadoop launched in 2006, the same year as AWS' cloud portfolio. HBase showed up in 2008. Many of the hiccups with running Hadoop and friends in containers or on cloud VM's boils down to how hostnames are resolved and advertised; not any significant design issue.

> Hadoop launched in 2006, the same year as AWS' cloud portfolio. Which makes it all the less reasonable to assume it would be in any way cloud native, when "the cloud" was at best a nascent idea at that point. And how many years did it take AWS to get any serious traction after launch?

[deleted]

Re: Kafka without ZooKeeper

#183

Post by cmckn »

pc86 wrote: post #155
cmckn wrote: post #64

Earlier quoted context omitted.

Hadoop launched in 2006, the same year as AWS' cloud portfolio. HBase showed up in 2008. Many of the hiccups with running Hadoop and friends in containers or on cloud VM's boils down to how hostnames are resolved and advertised; not any significant design issue.

> Hadoop launched in 2006, the same year as AWS' cloud portfolio. Which makes it all the less reasonable to assume it would be in any way cloud native, when "the cloud" was at best a nascent idea at that point. And how many years did it take AWS to get any serious traction after launch?

I haven't assumed anything of the sort, I said I loved the software and wished it was easier to use in what is considered a "modern" environment.

Re: Kafka without ZooKeeper

#184

Post by cmckn »

shaklee3 wrote: post #118
cmckn wrote: post #16

Earlier quoted context omitted.

Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) N…

> NATS doesn't ever store messages persistently Not true. Both Nats streaming and the upcoming jetstream (core nats) do.

I'm aware, I was referring to the core binary, `nats-server`. NATS streaming server seems to be still receiving attention, but the client library (for Java) hasn't been committed to since 2019, so I'm not sure I'd build a new project with it. JetStream is out (as of this week, I believe) and is an optional module to `nats-server`, as you said.

To the folks at Synadia -- I love NATS, but the naming and organization of these projects could use some work. What's with the `stan.*` repository names? Where did "jetstream" come from? Why is it baked into `nats-server` but `nats-streaming-server` isn't? Is `nats-streaming-server` on the back burner?

Re: Kafka without ZooKeeper

#185

Post by shaklee3 »

cmckn wrote: post #184
shaklee3 wrote: post #118

Earlier quoted context omitted.

> NATS doesn't ever store messages persistently Not true. Both Nats streaming and the upcoming jetstream (core nats) do.

I'm aware, I was referring to the core binary, `nats-server`. NATS streaming server seems to be still receiving attention, but the client library (for Java) hasn't been committed to since 2019, so I'm not sure I'd build a new project with it. JetStream is out (as of this week, I believe) and is an optional module to `nats-server`, as you said. To the folks at Synadia -- I love NATS, but the naming and organization of…

Stan is nats streaming. The clients don't have to be updated since they're forward compatible since jetstream. NATS streaming will be deprecated after jetstream is GA is my understanding. Is there a bug in the library you found?

Re: Kafka without ZooKeeper

#186

Post by travisjeffery »

chokolad wrote: post #171
travisjeffery wrote: post #139

Earlier quoted context omitted.

Hey, I'm the author of Jocko. I've been working at Confluent the past four years. I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-g...

Heh, I just bought it based on PragProg newsletter :). Literally 5 minutes ago.

Awesome! Hope you enjoy reading the book and thanks for buying it.

Re: Kafka without ZooKeeper

#187

Post by stalller »

incognito_limey wrote: post #141
jimmyed wrote: post #89

Earlier quoted context omitted.

Only you have to be a billionaire to use the confluent cloud.

So I guess pay as you go is too expensive for your org? Be curious to understand why you feel it's too expensive when it completely extrapolates any kafka management.

relative to more mature Kafka cloud services, it's crazy expensive.

Re: Kafka without ZooKeeper

#188

Post by agallego »

thibauts wrote: post #172
agallego wrote: post #162

Earlier quoted context omitted.

From a feature list perspective on the readme it basically is what we did at Redpanda (dev here)

Hi Alexander ! I'm a big fan of the work you do at vectorizedio and of your blog posts ! I strongly believe there's a need for much simpler event streaming and room for improvement performance-wise. I took a different path by divorcing from the Kafka protocol and experimenting with what I believe is a simpler to model approach to reliable event processing [1]. [1] https://github.com/dataptive/styx/blob/master/docs/ho…

that makes total sense. I agree. though in the future, the kafka proto will be impl details. i.e.: once you start moving to http proxy with typed schemas.

Re: Kafka without ZooKeeper

#189

Post by agallego »

sgt wrote: post #156
gigatexal wrote: post #154

Earlier quoted context omitted.

Doesn’t seem to be one. They took the Kafka API so it’s a drop in replacement and redid the implementation from the ground up. Your mileage may vary though.

I am just worries that things like 3rd parties integrating with your Kafka broker will stop working or experience issues. Also, I wonder if SASL_SSL is implemented exactly the same way etc.

hi @sgt - yup SASL_SSL due to the client protos have to be compatible, including the hack you are thinking of.

from a user perspective, existing SASL + SSL + SCRAM will be released next wednesday - so no code changes.

Re: Kafka without ZooKeeper

#190

Post by staticassertion »

mrkeen wrote: post #136
staticassertion wrote: post #97

Earlier quoted context omitted.

I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.

> You end up splitting your data across partitions anyways Messages are ordered within partitions. > What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle. TCP/IP ?

> Messages are ordered within partitions.

Right, but that means you're still "unordered" across those partitions?

> TCP/IP ?

But TCP/IP isn't delivered in order, it rearranges the unordered packages by their ID. I guess ordered delivery would be nice for that, but I just feel like making your protocol not require ordering is far simpler.

Not to mention that both TCP and Kafka have to handle head of line blocking?

I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons.

Post reply on HN