Pulsar vs. Kafka • Hacker News Board

Re: Pulsar vs. Kafka

#101

Post by jpgvm » Tue, Jul 14, 2020, 1:48 PM UTC

jarym wrote: post #29

I’ve enjoyed using Pulsar but ZooKeeper... arghh. It’s an excellent component but a pain to manage. Looking forward to trying Kafka again when they finally remove ZK

What difficulties have you had with Zookeeper and Kafka? Zookeeper can be difficult mostly because developers don't understand it very well. But in the case of Pulsar/BookKeeper/Kafka the usage of Zookeeper is very minimal so it's main problem constraint (performance) is mostly mitigated. Availability and management wise Zookeeper 3.5+ is actually pretty great. You do need to understand dynamic ensemble management but really it's a small price to pay for it's rock solid nature. Stuff like etcd is getting close these days but it took 3 protocol versions and tons of bugs, performance and scalability problems for it to get close to ZK.

Re: Pulsar vs. Kafka

#102

Post by doonesbury » Wed, Jul 15, 2020, 2:10 PM UTC

miguno wrote: post #96

doonesbury wrote: post #93

Earlier quoted context omitted.

On the issue of delivery ... can Pulsar users here talk about exactly once message delivery from the consumer side? I use Kafka and needed a cache based on incoming events which were partitioned. But if the consumer crashes it's not easy to pick up from exactly where it left off without manually committing offsets which hurts performance. There's also some hand waving Kafka gossip that it's hard to commit offsets rig…

Unlike Kafka, and despite some unfortunately misleading wordings in articles and documentation pages, Pulsar doesn't actually support exactly-once aka 'effectively-once' semantics because it lacks support for transactions. It only supports an idempotent producer combined with message deduplication. The current functionality only works when producing one message and to only one partition. For example, you cannot atomi…

Thanks. I'm gonna recheck Kafka then.

Re: Pulsar vs. Kafka

#103

Post by miguno » Wed, Jul 15, 2020, 2:50 PM UTC

doonesbury wrote: post #93

polskibus wrote: post #2

Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).

On the issue of delivery ... can Pulsar users here talk about exactly once message delivery from the consumer side? I use Kafka and needed a cache based on incoming events which were partitioned. But if the consumer crashes it's not easy to pick up from exactly where it left off without manually committing offsets which hurts performance. There's also some hand waving Kafka gossip that it's hard to commit offsets rig…

PS: Not fully sure what could have caused your Kafka woes. Certainly all what you described is supported, and it also 'should' normally be easy to use as a user/developer.

For example, with Kafka Streams, any app you build with it just needs to set “processing.guarantee” to “exactly_once” in its configuration, and regardless of what happens to the app or its environment it will not lose messages (on write) or miss messages (on read) from Kafka.

Consider asking your question with a few more details in the Kafka user mailing list [1], or in the Confluent Community Slack [2] if you prefer chatting.

[1] https://kafka.apache.org/contact [2] https://launchpass.com/confluentcommunity