41–50 of 201 posts
Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D
What do you use instead?
If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....
If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
Earlier quoted context omitted.
Still better than the sudo logo.
Wait.. sudo has a logo? Wow, that logo is ... special.
Wow indeed... I assume that’s a play on “sudo make me a sandwich”?
Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.
If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
They contribute heavily to the project and offer enterprise support, professional services, and proprietary tech like Cluster linking.
Source: I am an enterprise customer of Confluent.
If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
Hijacking seems like a strong word since Jay Kreps is the CEO.
Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible and I don’t see any reason to trust them as a company or pay for their software.
I’ve been sticking to pulsar for all new projects and have yet to hit a single drawback. It scales better, has less fiddly knobs needing adjusting, has cluster management already built in, and supports traditional pub/suv as well as worker queue semantics. It even has Kafka compatible adapters so it’s relatively easy to migrate existing systems.
Kafka played an important role in the history of distributed system design but it’s time to move to something better built and better managed IMO.
If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
This is a very uncharitable take on Confluent. They contribute heavily to the project and offer enterprise support, professional services, and proprietary tech like Cluster linking. Source: I am an enterprise customer of Confluent.
Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20
Using Kafka only for ordered pub-sub is like ordering tap water at French Laundry.
If you went there for tap water, yeah, maybe there are better options.