Apache Pulsar is an open-source distributed pub-sub messaging system • Hacker News Board

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#211

Post by JensRantil » Fri, Jan 03, 2020, 9:09 AM UTC

bauerd wrote: post #203

liquidgecka wrote: post #129

Earlier quoted context omitted.

Nats is not a replacement for pulsar or rabbitmq. It is a message passing system designed to pass lots of messages live, however if nobody is their to receive them they are lost and gone forever. There is a streaming layer but that is closer to Kafka and still does not provide the typical message model with an ack/nack API. I have used nats in several different ways but since it can be lossy its never been considered…

Sounds like zeromq to me

Semi-correct observation. You could think of it as a ZeroMQ broker that supports the pub/sub patternz, yes. Difference is it's centralized (with support for HA), adds security, gives you monitoring, supports multi-tenancy and horizontal scaling allowing to connect multiple clusters. Benefit of using broker is that service discovery is a no-op.

Also, NATS has client libraries for many languages that adds request/response semantics on top of the pub/sub semantics.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#212

Post by NicoJuicy » Fri, Jan 03, 2020, 9:28 AM UTC

_frkl wrote: post #56

tyri_kai_psomi wrote: post #46

Earlier quoted context omitted.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

Outbox pattern and NATS streaming

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#213

Post by JensRantil » Fri, Jan 03, 2020, 10:06 AM UTC

FridgeSeal wrote: post #177

atombender wrote: post #138

Earlier quoted context omitted.

I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.

And I believe Kafka has a KIP in progress to remove their reliance on Zookeeper altogether but having the brokers communicate with each other directly.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#214

Post by JensRantil » Fri, Jan 03, 2020, 10:08 AM UTC

NicoJuicy wrote: post #212

_frkl wrote: post #56

Earlier quoted context omitted.

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

Outbox pattern and NATS streaming

Additionally, you could always have reliable delivery over NATS using the request/response pattern using acking and retries.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#215

Post by JensRantil » Fri, Jan 03, 2020, 10:10 AM UTC

manigandham wrote: post #50

bovermyer wrote: post #12

How does this compare with NATS?

NATS is ephemeral pub/sub only. There is no persistence or replay, but focuses on high performance and messaging patterns like request/reply. Kafka and Pulsar persist every message and different consumers can replay the stream from their own positions. Pulsar also supports ephemeral pub/sub like NATS with a lot more advanced features. NATS does have the NATS Streaming project for persistence and replay but it has sca…

"Scalability issues" makes it sound like a blocer. I'd rephrase is as "doesn't support horizontal scaling which might be a limit to some".

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#216

Post by jsilence » Fri, Jan 03, 2020, 11:32 AM UTC

Would Pulsar be suitable for IoT messaging? An alternative to mqtt?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#217

Post by NicoJuicy » Fri, Jan 03, 2020, 11:33 AM UTC

mbostleman wrote: post #128

This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receive…

Look up : outbox pattern

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#218

Post by ypcx » Fri, Jan 03, 2020, 11:51 AM UTC

happythought wrote: post #200

ypcx wrote: post #78

Earlier quoted context omitted.

Alrighty, a few questions: - what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch? - have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly? - how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery - for the…

Do you have a link to the eks networking bug?

AWS was sending "ACTION REQUIRED" email to all EKS users about this. The Github issue is: https://github.com/aws/amazon-vpc-cni-k8s/issues/641

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#219

Post by [deleted] » Fri, Jan 03, 2020, 1:22 PM UTC

addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

[deleted]

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#220

Post by tylertreat » Fri, Jan 03, 2020, 3:54 PM UTC

omegabravo wrote: post #207

tylertreat wrote: post #130

Earlier quoted context omitted.

Check out Liftbridge ( https://liftbridge.io ) as a way to add these capabilities to NATS.

your FAQ still says it's not production ready? Is this still the case, I've been keeping my eye on this project

It's getting very close. I had wanted to make a production-ready 1.0 release before the end of the year, but we're in the process of switching from protobuf to flatbuffers. Once that is complete, a stable release will be made.