Earlier quoted context omitted.
Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?
Confluent Cloud has some nice point-and-click UI for creating associated Kafka resources like Schema Registries and Connect Clusters. My preference is MSK but I'm very comfortable with vanilla Kafka in AWS at a good price with auto-updates.
Kafka without ZooKeeper
81–90 of 201 posts
Re: Kafka without ZooKeeper
#82Setting up a Zookeeper ensemble is not that hard, they're light on resources and and basically zero maintenance.
Re: Kafka without ZooKeeper
#83If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client suppor…
No. No it doesn’t. It has at-least-once delivery with client-side deduplication. That’s not new, it’s what TCP does FFS. Why would you lie to people about supporting something long established at best and demonstrably impossible at worst?
> Finally, I work at Confluent....
Oh, that’s why. Never mind then. Continue selling digital snake oil.
Re: Kafka without ZooKeeper
#84Earlier quoted context omitted.
I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client suppor…
What challenges do you see Pulsar having (and potentially not overcoming)?
Re: Kafka without ZooKeeper
#85Earlier quoted context omitted.
You can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.
Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?
Re: Kafka without ZooKeeper
#86Earlier quoted context omitted.
Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to man…
Your clarification made me wonder: Is the single process deployment only doable via a container? Or will we actually have OS native process as well?
Re: Kafka without ZooKeeper
#87If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
More moving parts. Brokers, and Bookies, and ZK, plus proxies etc. Plus an additional ZK for inter-cluster replication.
Immaturity - it's still early days for Pulsar, and there's still a lot of bugs being found - and then rapidly fixed, full credit to them, but yeah, not yet as stable. Documentation is often obsoleted, and I found myself having read the code to figure out what was actually going on.
More complex workflows - there's only really one model for a developer consuming or producing against Kafka. With Pulsar, there's multiple different subscription modes, and choosing the wrong one could produce problems.
Also, the need to explicit ack the messages is something you'd have to always watch for to avoid duplicated reads. Also, if using batch receive, when I was looking at Pulsar, you either had to acknowledge the entire batch, or none of it, so a failure during batch processing would lead to the batch being reprocessed, but I think acking within a batch is in development.
No Pulsar IO S3 sink yet.
That said, there's a lot of cool things it's doing, like the built-in schema registry and far easier multitenancy, and offloading older data into S3 etc. transparently to the consumers, so I'm definitely I'm keeping an eye on it.
Lastly, you're taking aim at Confluent, you realise Pulsar is largely controlled by people employed by StreamNative, yeah?
Re: Kafka without ZooKeeper
#88Earlier quoted context omitted.
I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client suppor…
> The broader Kafka ecosystem has features like exactly-once support No. No it doesn’t. It has at-least-once delivery with client-side deduplication. That’s not new, it’s what TCP does FFS. Why would you lie to people about supporting something long established at best and demonstrably impossible at worst? > Finally, I work at Confluent.... Oh, that’s why. Never mind then. Continue selling digital snake oil.
Confluent make big bold claims "Exactly once delivery" and have aggressive marketing.
Pulsar on the other hand would say we have "effectivley-once". Reading Pulsar docs vs Kafka, Pulsar are very modest about functionality and have no commercial marketing at all.
These days I have noticed Confluent in blog posts do use effectively once but marketing is as aggressive as ever.
Credit where credit is due. Confluent, the marketing and big bold claims is why almost everyone is using Kafka and not Pulsar and may not of even heard of Pulsar. I do find Pulsar architecture more interesting, since Splunk has brought them though it's remained in the background like it always has with no huge push to sell it.
Re: Kafka without ZooKeeper
#89Earlier quoted context omitted.
Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) N…
you also don't have to worry about those kinds of configuration gotchas if you use confluent cloud!
Re: Kafka without ZooKeeper
#90IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.