Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pulsar vs. Kafka

streamnative.io

Post reply on HN
Print view View original

81–90 of 103 posts

Re: Pulsar vs. Kafka

#81

Post by atomicity »

throwaway_pdp09 wrote: post #76
nikhilsimha wrote: post #72

Earlier quoted context omitted.

Nightly to seconds is definitely great. Seconds to milliseconds is what is questionable I think.

> Nightly to seconds is definitely great Why? Can a business mobilise in anything less than days? If a report is minutes out of data, is that any loss? Given that some largish proportion of reports are never used, perhaps better management is key. not disagreeing but efficiency is not just a matter of quickness.

Nightly to seconds = developers can see the data generated within seconds.

This generally has implications on data quality, because you aren't fixing data quality issues once they occur in prod.

This also makes it far easier to develop ETL pipelines. You don't need complex tooling to see whether your ETL pipeline works.

You can technically fix data quality and dev velocity issues without low data freshness, but a quick glance at the data engineering landscape tells you they aren't being solved enough.

Re: Pulsar vs. Kafka

#82

Post by majidazimi »

kevstev wrote: post #78
majidazimi wrote: post #67

Earlier quoted context omitted.

That's true but still limitation is not fully resolved. In order to increase consumption rate, we need to add replicas. In pulsar Brokers are merely cache nodes over Bookkeeper. Adding more Brokers is trivial in Pulsar.

How in pulsar do they get around the fact that adding a new broker, data needs to be moved over before that broker can start serving data? This seems like a basic law of physics type limitation to me.

Network is faster than disk. Once cached, then you are only bound by network IO for subsequent uses.

Re: Pulsar vs. Kafka

#83

Post by kevstev »

majidazimi wrote: post #82
kevstev wrote: post #78

Earlier quoted context omitted.

How in pulsar do they get around the fact that adding a new broker, data needs to be moved over before that broker can start serving data? This seems like a basic law of physics type limitation to me.

Network is faster than disk. Once cached, then you are only bound by network IO for subsequent uses.

Sure- but how is this different than kafka's caching?

Re: Pulsar vs. Kafka

#84

Post by addisonj »

SmooL wrote: post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

Pulsar offers a standalone mode, which runs everything inside a single JVM and you can just backup the data files. Pulsar also can be made to run with multiple components, but not distributed.

Re: Pulsar vs. Kafka

#85

Post by addisonj »

akerro wrote: post #11
polote wrote: post #3

All those articles of kafka vs pulsar are always biased (this one is from a company selling pulsar). There are so many of them that I can't get an opinion on which one is good for what.

Pulsar is more flexible and fault-tolerant. For me the most important thing is client can request log queue starting from specific log by id, it has better retrying mechanisms for logs that failed to be processed. But it has absurdly bad documentation. I had to learn many things about Pulsar by downloading src of their java library and just reading the code. Documentation on starting bookie, zookie, pulsar, pulsar-pr…

This is definitely an area where Pulasr is trying to improve, getting started is not easy. That said, the progress I have seen in the last year since I have been involved with it is really promising.

Re: Pulsar vs. Kafka

#86

Post by tomkwong »

vp8989 wrote: post #30

AWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems). It's not all hype either, according to this post https…

How is Kinesis crappy?

Re: Pulsar vs. Kafka

#87

Post by tomkwong »

tcbasche wrote: post #10

> two of the most favored messaging systems on the market Give me a break, I'd literally never even heard of Pulsar until this article popped up. Of all messaging systems I would have thought Kafka vs SQS, or even RabbitMQ at the very least

I love SQS but it isn't quite comparable to Kafka because it is poll-based. It's a solid queue but not suitable for streaming.

Re: Pulsar vs. Kafka

#88

Post by addisonj »

kevstev wrote: post #78
majidazimi wrote: post #67

Earlier quoted context omitted.

That's true but still limitation is not fully resolved. In order to increase consumption rate, we need to add replicas. In pulsar Brokers are merely cache nodes over Bookkeeper. Adding more Brokers is trivial in Pulsar.

How in pulsar do they get around the fact that adding a new broker, data needs to be moved over before that broker can start serving data? This seems like a basic law of physics type limitation to me.

Hey, I work on Pulsar, will try and answer this :)

Topics (actually bundles of topics, called bundles) are what is assigned to Brokers. Topic assignment is dynamic, so when a new broker is added, the system will try and shed load from the busiest brokers to even it out on the system.

But unlike Kafka, when a topic is assigned to a broker, it doesn't have much state to move, mostly it just gets metadata added to it and opens a new "ledger" (which is just a chunk of the topics data over a time window, only one ledger is ever open at once). When it needs to serve data, it pulls that from bookkeeper nodes from previous ledgers, so the process of re-distributing load is pretty quick, it also doesn't eagerly pull in a cache.

Now, as far as the cache, that is primarily for "tailing reads", meaning, as writes occurs, and clients who are close to the tip of the recent data will just get it from the broker, without a need to pull it from bookkeeper. This is is one of the key parts about how Pulsar has multiple tiers of storage that help it have such good consistent latency.

Beyond processing writes, the biggest thing brokers do is handling "tailing reads" i.e., clients are consuming right near the tip of the topic. , this is the cache referred to. That means that when a new pbroker is three purposes:

1. Handling writes

Re: Pulsar vs. Kafka

#90

Post by vp8989 »

tomkwong wrote: post #86
vp8989 wrote: post #30

AWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems). It's not all hype either, according to this post https…

How is Kinesis crappy?

I don't think the platform or the pricing model were well architected for a 1 producer, many (and growing) consumer use case which is IMO the most compelling use case for a streaming system.

All the "success cases", sample architectures and real in the wild systems I've seen built on top of Kinesis have 1-3 consumers max.

They added the enhanced fan-out to try to get around this but it seems like you have to (over)pay for a ton of provisioned capacity to ensure you get decent latency on a > 5 consumer use case. So much like Lambda, it's only "serverless" or non-provisioned for people who have super low expectations of what system software ought to be capable of. For everyone else, it's just a mediocre expensive provisioned solution.

Post reply on HN