Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

111–120 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#111

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Not questioning your judgement but interested to know about the factors moving you away from Kinesis.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#112

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

The first question I have is why? SQS seems like such a simple thing to keep hosted.

They said they are currently doing 50k messages a second and they aren't even done migrating everything over.

50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check).

Plus, with sqs, you get what they have. No customizations.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#113

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Why did you choose Pulsar?

The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more flexible than any hosted options.

For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard.

Similarly, SQS does pub/sub well, but we keep finding that we do need to use the data more than the first initial delivery. Instead of having multiple systems where we store that data we have one.

As for why we didn't go with Kafka, the biggest single reason is that Pulsar is easier operationally with no needing to re-balance and also with the awesome feature that is tiered storage via offloading that allows us to actually do topics that have unlimited retention. Perhaps more importantly for the adoption though is pub/sub is much easier with Pulsar and the API is just much easier to reason about for developers than all the complexity of consumer groups, etc. There are a ton of other nice things like being able to have topics be so cheap such that we can have hundred of thousands and all of the built-in multi-tenancy features, geo-replication, flexible ACL system, pulsar functions and pulsar IO and many other things that really have us excited about all the capabilities

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#114
post #99

Earlier quoted context omitted.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are…

Is it me or does NATS looks like it's aimed at an actor-based style of distributed system ?

It not necessarily aiming at that.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#115

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

We provide a SaaS offering of Apache Pulsar in AWS, Azure, and GCP: https://kafkaesque.io/

Cool name. That's one of those company names that almost seems like someone thought it would make a good company name first and thought it was so fitting, they should build a company around it.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#116

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

The first question I have is why? SQS seems like such a simple thing to keep hosted.

Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cases, it is being able to have one tech that does SQS like stuff and Kinesis like stuff in one place

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#117

Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead. Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet. CS is damned, and this is hell.

not really, people legitimately need this for truly big data. you can't reliably processing 7 trillion events per day with a completely C/unix CLI stack.

I would bet that vast, vast majority of production kafka deployments do not see 7 trillion events per day. I bet many/most do not even see 7 billion events per day.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#118
post #78

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Alrighty, a few questions: - what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch? - have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly? - how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery - for the…

Oh forgot one aspect about DR: for critical data, we can easily turn on geo-replication (with a single API call) and have that data now in another region purely for DR purposes (or for cross region use cases)

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#119
post #79

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Really interested why you chose Pulsar over RabbitMQ and others?

I have used (and deployed) rabbitmq in the past and really love it for pub/sub, but for our needs, we keep needing retention, particularly long retention that we process with Flink for computing views. Having one system to do both is great for us.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#120
post #111

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Not questioning your judgement but interested to know about the factors moving you away from Kinesis.

Kinesis is very expensive in the long run. There's almost always an intersection point on AWS where you need to consider moving away from AWS services/"managed services" and bring it in house.
Post reply on HN