Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pulsar vs. Kafka

streamnative.io

Post reply on HN
Print view View original

61–70 of 103 posts

Re: Pulsar vs. Kafka

#61

Post by toomanybits »

sciurus wrote: post #38
ramraj07 wrote: post #19

Earlier quoted context omitted.

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design…

If you can't understand what Zookeeper is, I'd recommend reading Martin Kleppmann's book Designing Data-Intensive Applications ( https://dataintensive.net/ ). You don't need a CS degree to work in this field (I don't have one either!) but there are fundamental concepts you need to understand in order to make informed decisions when designing distributed systems.

You don't need a CS degree or Martin Kleppmann's book to work out it's a GPITA.

Re: Pulsar vs. Kafka

#62

Post by Jedd »

toomanybits wrote: post #53

This smacks of being heavily one-product-focussed to me. Being a Kafka user it's hard enough managing and understanding one system, nevermind three or four joined together. Maybe it's a bit faster or a bit more elastic, or whatever, who knows. What I really care about is whether I get called at 3am and in that regard the argument seems pretty weak. Kafka for all its woes is a solid system you know you can count on. I…

What kind of (lower level) surrogate metrics would you be interested in that could translate to '3am phone calls' when comparing messaging systems?

Re: Pulsar vs. Kafka

#63

Post by Jedd »

polote wrote: post #32
maps7 wrote: post #31

Earlier quoted context omitted.

> Yet pulsar I only came across on HN a week back. Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.

If you search you will find that there are always the same 4-5 HN users who comment on pulsar/kafka topics arguing that Kafka is not a viable solution anymore. Even though almost everyone is still using Kafka

Almost everyone still uses Microsoft Windows.

But almost everyone has heard of GNU/Linux or Apple OSX.

Re: Pulsar vs. Kafka

#64

Post by [deleted] »

willvarfar wrote: post #33
doctor_eval wrote: post #25

> Lower end-to-end latency helps enterprises gain business insights faster. They lost me here. I can think of plenty of situations where reduced latency is beneficial, but not many situations where shaving a few milliseconds would make a difference to “business insight”! Although I suppose it is strictly correct, in the tautological sense...

So I've developed an in-house fancy thoroughbred 'real time' data warehouse, a very rare beast indeed, and its awesome. Of course, our business is still running on nightly reports. But the tech is cool! So I wanna say you're wrong, but I've got man-years invested in a system that hasn't been utilized to its full potential yet :( Time will tell. Somewhere, some competitor will be using real-time insights to out-compet…

[deleted]

Re: Pulsar vs. Kafka

#65

Post by itestyourcode »

polskibus wrote: post #2

Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).

Full disclosure - I am co-founder of https://kesque.com . We provide Pulsar as a managed service and different tiers of SaaS plan. My opinion can be biased.

For a new project, you should try to document different aspects of requirements. Is it data streaming, queuing, or both? What's the data retention policy? Message rate? How many consumers and producers? Any inbound or outbound integration with 3rd party destination (i.e. S3, Flink)? Both Kafka and Pulsar have so many features to offer. It is not a simple task to pick one vs another. If you ask for guaranteed delivery, both will satisfy that requirement. A level up question would be who can guarantee in-order delivery.

Managing Kafka and Pulsar require knowledge. I do not think any of these durable messaging software is maintenance free (or industry is not there yet). Any reliable distributed system is complex out of necessity. These system more or less require log consensus algorithm to achieve high availability. They all use either zookeeper or one of raft implementations requiring multiple nodes to perform leader election. This is common in all distributed architecture (kafka, Pulsar, Cockroach, etcd...). I would attest Pulsar can be administratively simpler than Kafka, because of separation of broker and bookkeeper (data persistent layer). But this does not mean any dev-op without knowledge can proficiently manage the cluster. We use Kubernets/Helm to manage all of our Pulsar clusters. I would not credit Pulsar alone with low operation upkeep. It is combinations of Kubernetes, Helm, in-house tools, and engineering knowledge to lower the operation cost.

Re: Pulsar vs. Kafka

#66

Post by toomanybits »

Jedd wrote: post #62
toomanybits wrote: post #53

This smacks of being heavily one-product-focussed to me. Being a Kafka user it's hard enough managing and understanding one system, nevermind three or four joined together. Maybe it's a bit faster or a bit more elastic, or whatever, who knows. What I really care about is whether I get called at 3am and in that regard the argument seems pretty weak. Kafka for all its woes is a solid system you know you can count on. I…

What kind of (lower level) surrogate metrics would you be interested in that could translate to '3am phone calls' when comparing messaging systems?

Being used by at least one company of significant size that (a) i've heard of and (b) isn't directly connected to the project would be a good start.

Re: Pulsar vs. Kafka

#67

Post by majidazimi »

toomanybits wrote: post #56
majidazimi wrote: post #21

Earlier quoted context omitted.

There are real differences among them. Here is some painful aspects of Kafka: 1. A single partition is stored in one node (replicas on another nodes). With this, introducing new nodes takes very long time to replicate large partitions, because it can replicate one partition from only one node (leader of the partition). On Pulsar each segment of partition is stored in a different bookkeeper node. 2. Because of 1, if t…

1. is true, but if you want that data to move to a new node, it still needs to be replicated. Kafka's approach is to use tiered storage (which I believe is close to completion). 2. Kafka can read from a replica node. It's relatively new but it's there.

That's true but still limitation is not fully resolved. In order to increase consumption rate, we need to add replicas. In pulsar Brokers are merely cache nodes over Bookkeeper. Adding more Brokers is trivial in Pulsar.

Re: Pulsar vs. Kafka

#68

Post by [deleted] »

majidazimi wrote: post #21
trengrj wrote: post #15

With Pulsar vs Kafka, I don't see a huge argument between either one functionality wise as they have so much in common (distributed log, Java based, avoid copying memory, use Zookeeper). Because Kafka is more supported and well-known it seems Pulsar needs to be an order of magnitude more performant to capture developer mindshare. I see the same with Spark vs Flink in that similarities outweigh differences. I wonder i…

There are real differences among them. Here is some painful aspects of Kafka: 1. A single partition is stored in one node (replicas on another nodes). With this, introducing new nodes takes very long time to replicate large partitions, because it can replicate one partition from only one node (leader of the partition). On Pulsar each segment of partition is stored in a different bookkeeper node. 2. Because of 1, if t…

[deleted]

Re: Pulsar vs. Kafka

#69

Post by papaf »

toomanybits wrote: post #61
sciurus wrote: post #38

Earlier quoted context omitted.

If you can't understand what Zookeeper is, I'd recommend reading Martin Kleppmann's book Designing Data-Intensive Applications ( https://dataintensive.net/ ). You don't need a CS degree to work in this field (I don't have one either!) but there are fundamental concepts you need to understand in order to make informed decisions when designing distributed systems.

You don't need a CS degree or Martin Kleppmann's book to work out it's a GPITA.

I am also not a fan of Zookeeper but I have come to respect it. In defense of Zookeeper, distributed systems are a GPITA. If you need to select some components why not go for ones that are solid [1].

[1] https://aphyr.com/posts/291-jepsen-zookeeper

Re: Pulsar vs. Kafka

#70

Post by kjeetgill »

itestyourcode wrote: post #54
doctor_eval wrote: post #25

> Lower end-to-end latency helps enterprises gain business insights faster. They lost me here. I can think of plenty of situations where reduced latency is beneficial, but not many situations where shaving a few milliseconds would make a difference to “business insight”! Although I suppose it is strictly correct, in the tautological sense...

I agree shaving dozens or hundreds milliseconds of latency is hardly noticeable to end users. But latency is an indicator of how well the system can perform and scale up. Signs of high latency under normal load can reveal design or implementation flaws in the software (supposedly running on any modern hardware) Ultimately you want a system can scale up and delivers consistent latency. Therefore, a low and consistent…

I agree with everything you're saying about the value of reduced/smoothed out latencies and I suspect the parent agrees with that as well, but the parent was mostly teasing about the choice of "business insights" in lieu of all you've mentioned.
Post reply on HN