Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

61–70 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#61

Post by vorpalhex »

tyri_kai_psomi wrote: post #46
geodel wrote: post #38

Earlier quoted context omitted.

This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java. Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

NATS is amazing but note that it makes different promises than Pulsar. NATS doesn't offer true durability (in exchange for amazing performance and great simplicity) whereas Pulsar and similar are meant to survive certain partition or failure situations and not lose data.

It's not one or the other, they're just different tools.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#62

Post by mattboyle »

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt feel we could do it and still develop the product. I'll continue to keep an eye on pulsar but for now Kafka is the clear go to imo. It's well documented, great SAS offerings (confluent) and tons of books and training courses for it.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#63

Post by tyri_kai_psomi »

_frkl wrote: post #56
tyri_kai_psomi wrote: post #46

Earlier quoted context omitted.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

In my thinking, I think of an endpoint as something at either end of the communication channel (NATS in this case) where it is effectively terminal. Usually this is where the application logic lies. Dereck Collison (creator of NATS) brings this up in many of his talks about NATS, but I think the source of his thinking might come from “End-to-End Arguments in System Design” by Saltzer, Reed, & Clark.

The core of it is this point:

"Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level."

That is, in order get that message redundancy or exactly once delivery, or message persistence, you pay a high cost, and you may be better off delegating to the endpoints.

This blog provides a good overview

https://blog.acolyer.org/2014/11/14/end-to-end-arguments-in-...

Here is the original paper

http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#65

Post by EGreg »

sz4kerto wrote: post #58
EGreg wrote: post #27

Earlier quoted context omitted.

What about ZeroMQ? Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn’t it far more performant and distributed? Maybe I am missing something here.

> Why use RabbitMQ and Kafka if you can use ZeroMQ? They are totally different, you're comparing apples with oranges. ZeroMQ gives you basic, very fast tooling to communicate between distributed processes. ZeroMQ does not provide tooling for e.g. maintaining a strictly ordered, multi-terabyte event log. And so on.

Yes but isn’t this a bit like comparing git / bitkeeper vs subversion / perforce?

Basically, one is decentralized and you can set up a massively parallel architecture, with eg each topic or subthread having its own pubsub.

The other is a monolithic centralized pubsub architecture.

You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn.

But for most use cases, ZeroMQ would allow far more flexible distributed systems topologies and solutions. No?

Edit: HN and Google are both awesome: https://news.ycombinator.com/item?id=9634925

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#66

Post by gperinazzo »

c0brac0bra wrote: post #19
tyingq wrote: post #7

Earlier quoted context omitted.

It exists: https://pulsar.apache.org/docs/en/next/client-libraries-node...

However it currently lacks the ability to listen for messages and run an event handler when one comes in: https://github.com/apache/pulsar-client-node/pull/56 You have to manually call ".receive()" to attempt to receive a message.

Using `.receive()` will occupy a worker thread from node until it returns. Having multiple consumers waiting on receive will clog up the worker threadpool, preventing anything that uses it from running. If you want to use the consumer right now, I would suggest always using a timeout on the receive call, and waiting between timed-out calls to receive. This is extremely important if you have multiple consumers.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#67

Post by [deleted] »

vorpalhex wrote: post #61
tyri_kai_psomi wrote: post #46

Earlier quoted context omitted.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

NATS is amazing but note that it makes different promises than Pulsar. NATS doesn't offer true durability (in exchange for amazing performance and great simplicity) whereas Pulsar and similar are meant to survive certain partition or failure situations and not lose data. It's not one or the other, they're just different tools.

[deleted]

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#68

Post by addisonj »

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS).

It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to get auth tokens for an AWS IAM role.

Overall, we are very pleased and the rest of the engineering org is very excited about it and planning to migrate most of our SQS and Kinesis apps.

Ask me anything in thread and will try and answer questions. At some point we will do a blog post on our experience.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#69

Post by ckdarby »

mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

> found the documentation very lacking

Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this?

> As a small startup

You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#70

Post by mindw0rk »

buboard wrote: post #57
Eikon wrote: post #55

Earlier quoted context omitted.

If you did bother to read the linked page, you would have understood that it’s a yahoo project handed over to Apache for management like many of Apache’s projects.

yeah i m talking more generally about their full list here: https://www.apache.org/

There are a lot of projects that were handed to Apache to manage. Kafka for example was initially created by LinkedIn. So yeah, you are right, big corps are actually creating those tools, and in addition to this, giving it away as open source to public.
Post reply on HN