Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pulsar vs. Kafka

streamnative.io

Post reply on HN
Print view View original

51–60 of 103 posts

Re: Pulsar vs. Kafka

#51

Post by itestyourcode »

SmooL wrote: post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

Pulsar does offer a standalone mode. It has a vertical stack of Pulsar broker, bookkeeper, and zookeeper in one process. The standalone mode also comes in a single docker image.

However, the complexity you refer to is essential in a reliable messaging framework. Use of zookeeper or any log consensus algorithm requires multiple nodes ( 3 or more) to achieve durability and high availability goal. It is out of necessity. This is actually essential complexity.

There is another messaging framework called Nats.io. It is not persistent so architecturally relatively simpler. You might want to investigate.

Re: Pulsar vs. Kafka

#52

Post by z9e »

qaq wrote: post #18
trengrj wrote: post #15

With Pulsar vs Kafka, I don't see a huge argument between either one functionality wise as they have so much in common (distributed log, Java based, avoid copying memory, use Zookeeper). Because Kafka is more supported and well-known it seems Pulsar needs to be an order of magnitude more performant to capture developer mindshare. I see the same with Spark vs Flink in that similarities outweigh differences. I wonder i…

Pulsar is better for very large scale deployments provided you have people to manage it

Kafka is handling very large scale deployments just fine atm in all the big tech co's.

The only thing I can see that can make this true is Pulsar seems to have better elastic scalability. But it seems to score less on everything else. It has a much more complex storage system that ends up not matching Kafka's high-end throughput at large scale.

From what I recall, Twitter ended up abandoning BookKeeper due to storage scale concerns. Related: https://blog.twitter.com/engineering/en_us/topics/insights/2...

Re: Pulsar vs. Kafka

#53

Post by toomanybits »

This smacks of being heavily one-product-focussed to me. Being a Kafka user it's hard enough managing and understanding one system, nevermind three or four joined together.

Maybe it's a bit faster or a bit more elastic, or whatever, who knows. What I really care about is whether I get called at 3am and in that regard the argument seems pretty weak. Kafka for all its woes is a solid system you know you can count on.

I'd much rather see someone come up with a truly innovative alternative that actually pushes the boundaries, rather than just copying what's there already, and adding a few window dressings.

Re: Pulsar vs. Kafka

#54

Post by itestyourcode »

doctor_eval wrote: post #25

> Lower end-to-end latency helps enterprises gain business insights faster. They lost me here. I can think of plenty of situations where reduced latency is beneficial, but not many situations where shaving a few milliseconds would make a difference to “business insight”! Although I suppose it is strictly correct, in the tautological sense...

I agree shaving dozens or hundreds milliseconds of latency is hardly noticeable to end users. But latency is an indicator of how well the system can perform and scale up. Signs of high latency under normal load can reveal design or implementation flaws in the software (supposedly running on any modern hardware) Ultimately you want a system can scale up and delivers consistent latency. Therefore, a low and consistent latency is a health-meter to assure that. Within the same cluster (no network I/O), Pulsar has pretty impressive 5ms pub/sub latency for persistent topics (including written to and acknowledged by the non-volatile disk)

Re: Pulsar vs. Kafka

#55

Post by cmorgan31 »

jmchuster wrote: post #4
polskibus wrote: post #2

Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).

If you are worried about operational complexity and don't know enough to get a clear set of requirement and tradeoffs beforehand, just start off with a single rabbitmq node. By the time you outgrow it, you'll have hired someone who'll be able to make the call for you.

This is likely the best advice in practice as the complexity kafka introduces really only becomes apparent at scale. We ran managed services for our primary production in the 50k+ RPS range that needed a constant stream of tweaking. On the other hand, the isolated EU cluster was self hosted and ran without incident or intervention for 18 months at 1-3k RPS.

Re: Pulsar vs. Kafka

#56

Post by toomanybits »

majidazimi wrote: post #21
trengrj wrote: post #15

With Pulsar vs Kafka, I don't see a huge argument between either one functionality wise as they have so much in common (distributed log, Java based, avoid copying memory, use Zookeeper). Because Kafka is more supported and well-known it seems Pulsar needs to be an order of magnitude more performant to capture developer mindshare. I see the same with Spark vs Flink in that similarities outweigh differences. I wonder i…

There are real differences among them. Here is some painful aspects of Kafka: 1. A single partition is stored in one node (replicas on another nodes). With this, introducing new nodes takes very long time to replicate large partitions, because it can replicate one partition from only one node (leader of the partition). On Pulsar each segment of partition is stored in a different bookkeeper node. 2. Because of 1, if t…

1. is true, but if you want that data to move to a new node, it still needs to be replicated. Kafka's approach is to use tiered storage (which I believe is close to completion).

2. Kafka can read from a replica node. It's relatively new but it's there.

Re: Pulsar vs. Kafka

#57

Post by toomanybits »

qaq wrote: post #18
trengrj wrote: post #15

With Pulsar vs Kafka, I don't see a huge argument between either one functionality wise as they have so much in common (distributed log, Java based, avoid copying memory, use Zookeeper). Because Kafka is more supported and well-known it seems Pulsar needs to be an order of magnitude more performant to capture developer mindshare. I see the same with Spark vs Flink in that similarities outweigh differences. I wonder i…

Pulsar is better for very large scale deployments provided you have people to manage it

Show me one

Re: Pulsar vs. Kafka

#59

Post by toomanybits »

ypcx wrote: post #17
polskibus wrote: post #2

Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).

The good. To have 100% reliable Kafka, your data (both transfer and storage) needs bot replicated by the factor of 3 (you will need to transfer and store 3 copies of all your data). Pulsar will happily do with just 2x. Next, thanks to the "failover" subscription type and producer real-time deduplication, high-availability consumers are easier/cleaner done on Pulsar. On Kafka, you have to fiddle with Consumer Groups a…

> Pulsar will happily do with just 2x. This is just wrong. Pulsar provides weaker guarantees than Kafka. It's a quorum based system. If you run with two replicas Pulsar can't provide F-1 guarantees which Kafka can.

Re: Pulsar vs. Kafka

#60

Post by toomanybits »

BenoitP wrote: post #12
polskibus wrote: post #2

Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).

One very specific feature where Pulsar is shining is that you don't need to explicitly create topics [1]. It doesn't seem like much but is very powerful. At least for a CQRS subscription architectural pattern I'm working on at the moment. Say you have a front-end dealing with the clients in a streaming manner (be it websockets or SSE). All front-end instances send messages to a topic on a messaging system. Processing…

I'm lost too. Kafka auto creates topics by default. Maybe you're referring to being able to create more topics? But that seems to be unproven. Kafka's limit is metadata and Pulsar is more metadata dependent than Kafka.
Post reply on HN