Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

91–100 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#91

Post by mattboyle »

ckdarby wrote: post #69
mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

> found the documentation very lacking Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this? > As a small startup You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.

It was about 6 months ago.

I completely disagree with the opex of picking up kafka vs developing a whole client library. Please could you try and explain how you came to this conclusion?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#92

Post by mpmpmpmp »

reggieband wrote: post #89

I'm still on the fence with these distributed log/queue hybrids. From a theoretical perspective it seems these are excellent. I just have this nagging suspicion that there is some even-worse problem architectures based on these systems will harbor. This kind of ambivalence is something I find myself having to battle more and more in my career as I age. Most of the time the hype around new design/development patterns…

I totally get where you're coming from as I feel like that alot too. But the fact that you are thinking about risk and business value over working on cool new tech should be a positive for the ventures you are a part of.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#93

Post by pm90 »

bsaul wrote: post #18

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…

It depends. We have a ton of Java apps running atop of kubernetes. All of them use zk, but every team operates their own mini zk cluster deployed on k8s. It’s worked fine except for certain hard to debug problems that hit a few teams occasionally.

I guess my point is that k8s let’s you shift the operational burden to dev teams if they need it. If you have a centralized operations team running a giant, common zk/ etcd, yeah this would be additional operational burden.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#94

Post by matteomerli »

mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

We're close to release a new "officially supported" native Go client library: https://github.com/apache/pulsar-client-go

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#95

Post by jjjensen90 »

vorpalhex wrote: post #61
tyri_kai_psomi wrote: post #46

Earlier quoted context omitted.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

NATS is amazing but note that it makes different promises than Pulsar. NATS doesn't offer true durability (in exchange for amazing performance and great simplicity) whereas Pulsar and similar are meant to survive certain partition or failure situations and not lose data. It's not one or the other, they're just different tools.

There is nats-streaming-server as well which offers true durability (via file or SQL store) and a streaming model very similar to Kafka and Pulsar. It can also run as a raft cluster or in fault tolerance mode. It still has very good performance and is very simple to deploy and operate (I use it for event sourcing for real time IoT data at my day job).

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#96

Post by jgraettinger1 »

mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

If you're a Go shop, Gazette is worth a look (https://gazette.dev).

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#97

Post by richdougherty »

cbartholomew wrote: post #47
TuringNYC wrote: post #40

Earlier quoted context omitted.

I went to https://pulsar.apache.org but didnt find a "Why Pulsar and not Kafka" -- is there an answer to that, or is this another Kafka competitor with the same strengths and not a specific differentiator?

Here is a two-part blog post I wrote on why Pulsar and not Kafka: https://kafkaesque.io/5-more-reasons-to-choose-apache-pulsar...

Thanks! And Part 1 seems like a good place to start: https://kafkaesque.io/7-reasons-we-choose-apache-pulsar-over...

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#98

Post by gen220 »

cpx86 wrote: post #76
ckdarby wrote: post #69

Earlier quoted context omitted.

> found the documentation very lacking Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this? > As a small startup You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.

> You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar. Could you elaborate why this would be the case?

Not the OP, but I think they were exaggerating a bit. In practice, operating kafka is a major PITA, because it means you have to

(1) choose a "flavor" wrapper (confluent seems to be a popular one), because the base project isn't easy to develop against

(2) write your own wrappers of those wrappers, to keep your developers from shooting themselves in the foot with wacky defaults

(3) suffer the immense pain that is authenticating topic write/reads, if that's even possible???

(4) stand up zookeeper... and probably lose some data along the way.

(5) suffer zookeeper outages due to buggy code in kafka/zk (I've experienced lost production data due to unpredictable bugs in kafka/zk, but obviously YMMV).

Based on my naive assessment, the kafka/zookeeper ecosystem is maybe 10x as complicated as the problem it's solving, and that shows up in the OpEx. I personally doubt that Pulsar is that much better, but it might be.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#99

Post by bsaul »

tyri_kai_psomi wrote: post #46
geodel wrote: post #38

Earlier quoted context omitted.

This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java. Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

Is it me or does NATS looks like it's aimed at an actor-based style of distributed system ?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#100

Post by addisonj »

ypcx wrote: post #78
addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Alrighty, a few questions: - what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch? - have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly? - how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery - for the…

for the k8s definitions, we started with the helm chart, rendered the template, and then moved it into kustomize, as that is our tool of choice ATM, IDK if I would recommend that approach for everyone (we expect we might move to helm v3 at some point) but it was a good choice for us.

We have practiced some disaster recovery, but it isn't 100% exhaustive (is it ever?), however it is also aided by how Pulsar is designed. We have killed bookie nodes as well as lost all our state in zookeeper. The first is pretty easily handled by the replication factor of bookkeeper data and for zookeeper we do extra backup step and just dump the state to s3 and can restore it. What we haven't tested in practice but now how to do theoretically is to restore a k8s stateful set from EBS volume snapshots. However, we see that as a real edge case. In Pulsar, we offload our data to s3 after a few hours, so we only need to worry about potentially losing a few hours of data in BK, as the zookeeper state is very easy to just snapshot and restore from s3. In other words, we are still working on getting more and more confident with data and don't yet recommend teams use it for mission critical non-recoverable data, but there are a ton of uses cases for it now and we can continue to improve on the DR front

We have done multiple upgrades and deploy all the time. Because bookkeeper nodes are in a stateful set and we have don't do automated rollouts, we manually have a process to replace the BK nodes. However, they don't trigger a re-balance as it closes gracefully and then re-attaches the EBS volume from the stateful set

We use EBS volumes, we use a piops volumes for the journal and a larger slower volume for the ledger store. THis is one of the great parts of bookkeeper design is that the two disks pools are separate so we just need a small chunk of really fast storage and then the journaled data is copied over to the ledger volume by a background process. We figure for really high write throughput we could use instance storage for the journal volume and EBS for ledger, but that would have some complications on recovery but still easier than having to rebuild the whole ledger data.

We use the pulsar proxy and expose it via a k8s service with the AWS specific NLB annotations.

We haven't had any issues with the k8s plugin and haven't really had any issues with EKS version upgrades. We just add new nodes when we migrate the kubelets

Yes, we have automation (via terraform) to allow us to add many different pools of compute and we use labels and taints to get specific apps mapped to specific pools of compute. For Pulsar, we run all the components multi-AZ

Post reply on HN