Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

71–80 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#71

Post by _frkl »

tyri_kai_psomi wrote: post #63
_frkl wrote: post #56

Earlier quoted context omitted.

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

In my thinking, I think of an endpoint as something at either end of the communication channel (NATS in this case) where it is effectively terminal. Usually this is where the application logic lies. Dereck Collison (creator of NATS) brings this up in many of his talks about NATS, but I think the source of his thinking might come from “End-to-End Arguments in System Design” by Saltzer, Reed, & Clark. The core of it is…

Thanks, much appreciated!

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#72

Post by ckdarby »

srameshc wrote: post #64

This is great !! What would be the easiest way to run a 3 node cluster ?

The standalone mode will let you get started as a developer. You grab tar.gz, uncompress, run standalone.sh.

There are helm charts for running an actual cluster.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#73

Post by rocky1138 »

addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Why did you choose Pulsar?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#74

Post by geodel »

_frkl wrote: post #56
tyri_kai_psomi wrote: post #46

Earlier quoted context omitted.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

Did you check this?

https://docs.nats.io/nats-streaming-concepts/intro

"..Message/event persistence - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations."

and

"At-least-once-delivery - NATS Streaming offers message acknowledgements between publisher and server (for publish operations) and between subscriber and server (to confirm message delivery). Messages are persisted by the server in memory or secondary storage (or other external storage) and will be redelivered to eligible subscribing clients as needed."

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#75

Post by bubbleRefuge »

addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Isn't using Kubernetes kind of an anti-pattern due to failover and rebalancing logic clashing? If Kubernetes is killing and re-starting nodes and the cluster's brokers are detecting dead brokers and rebalancing partitions as a result, it seems counterproductive.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#76

Post by cpx86 »

ckdarby wrote: post #69
mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

> found the documentation very lacking Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this? > As a small startup You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.

> You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.

Could you elaborate why this would be the case?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#77

Post by whycombin8 »

matteomerli wrote: post #20
bsaul wrote: post #18

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…

> is pulsar going to take the same road ? Yes, it's in the works

Any plans for bookkeeper?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#78

Post by ypcx »

addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Alrighty, a few questions:

- what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch?

- have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly?

- how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery

- for the Bookies, do you use AWS EBS volumes with the EKS or just local instance storage (that is, if you use persistent topics)

- do you use the Proxy pod's EKS k8s pod IPs as exposed on the AWS network, or do you use a NodePort type of service for the Proxy components (using the EKS node IPs)

- have you been bitten by the recent EKS k8s network plugin bug (loss of pod connectivity), and/or how do you maintain your EKS cluster

- do you run your EKS nodes in a multi-AZ setting?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#79

Post by GordonS »

addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Really interested why you chose Pulsar over RabbitMQ and others?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#80

Post by rhizome »

bubbleRefuge wrote: post #75
addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Isn't using Kubernetes kind of an anti-pattern due to failover and rebalancing logic clashing? If Kubernetes is killing and re-starting nodes and the cluster's brokers are detecting dead brokers and rebalancing partitions as a result, it seems counterproductive.

To the degree that that conflict exists in their implementation I would think that it's possible to account for all of that.
Post reply on HN