Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

161–170 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#161

Post by alfalfasprout »

I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry.

Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support complex partitions, extreme fault tolerance (instead of more clever recovery logic), etc. add a lot of overhead. To the point of talking about "low latency" overhead in the order of 5ms instead of microseconds or even nanoseconds as is expected in trading.

Worse, many startups try to adopt these technologies where their message rates are miniscule. To give you some context, even two beefy machines with an older message queue solution like ZeroMQ can tolerate throughput in excess of what most companies produce.

This is not to discredit the authors of Pulsar or Kafka at all... but it's just a concerning trend where easy to use horizontally scalable message queues are being deployed everywhere. Similar to how everyone was running hadoop a few years back even when the data fit in memory.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#162

Post by gfodor »

alfalfasprout wrote: post #161

I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry. Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support comp…

Worth noting that Kafka is not a queue, but an append-only log.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#163

Post by ignoramous »

addisonj wrote: post #142
ignoramous wrote: post #136

Earlier quoted context omitted.

On behalf of everyone here, thanks a lot for answering every single question being asked. Highly appreciate it. I have questions myself: 1. Did it reduce (TCO) costs or increase it versus using Kinesis and SQS/SNS? 1a. Interestingly, there's no global-replication with those AWS services. Why did you require global-replication with the move to Apache Pulsar? 2. Since you mention internal auth : Weren't Cognito / KMS /…

1. On a short time horizon, not as sure, back of the napkin, it took ~12 dev months (5 months with 2.5 people average on it). However, our cost per 1000 msgs/sec is much lower (like 1/4 the cost of Kinesis) so we fully expect that investment to pay off over time assuming that adoption by the rest of the org continues and we don't find a ton of issues. 1a. You are correct we didn't require geo-replication for existing…

Agreed.

One more:

For replication across regions, do you peer VPCs via Transit Gateways or some such, or do it over the public Internet? I ask because a lot of folks complain about exorbitant AWS bandwidth charges for cross-AZ and cross-region communication (esp over the Internet versus over AWS' backbone): At 300k msgs/sec, the bandwidth costs might add up quickly?

Consequently, maintaining a multi-region, multi-AZ VPC peering might have been complicated without Transit Gateway, so I'm curious how the network side of things held up for you.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#164

Post by matteomerli »

wilhow wrote: post #149
matteomerli wrote: post #20

Earlier quoted context omitted.

> is pulsar going to take the same road ? Yes, it's in the works

Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?

That's correct, we're moving to have a pluggable metadata store and coordination service.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#165

Post by skube »

bubbleRefuge wrote: post #75
addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Isn't using Kubernetes kind of an anti-pattern due to failover and rebalancing logic clashing? If Kubernetes is killing and re-starting nodes and the cluster's brokers are detecting dead brokers and rebalancing partitions as a result, it seems counterproductive.

Using kubernetes is always an anti-pattern.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#166

Post by addisonj »

ignoramous wrote: post #163
addisonj wrote: post #142

Earlier quoted context omitted.

1. On a short time horizon, not as sure, back of the napkin, it took ~12 dev months (5 months with 2.5 people average on it). However, our cost per 1000 msgs/sec is much lower (like 1/4 the cost of Kinesis) so we fully expect that investment to pay off over time assuming that adoption by the rest of the org continues and we don't find a ton of issues. 1a. You are correct we didn't require geo-replication for existing…

Agreed. One more: For replication across regions, do you peer VPCs via Transit Gateways or some such, or do it over the public Internet? I ask because a lot of folks complain about exorbitant AWS bandwidth charges for cross-AZ and cross-region communication (esp over the Internet versus over AWS' backbone): At 300k msgs/sec, the bandwidth costs might add up quickly? Consequently, maintaining a multi-region, multi-AZ…

In this case, we use just straight VPC peering with a full mesh of all our regions. We may eventually migrate to being built on our VPN based mesh (we do that in other places)

Bandwidth is certainly a concern and that is one of the nice bits about Pulsar is not everything is replicated. You mark a namespace by adding additional clusters it should replicate to. We don't expect to replicate everything, just the things teams care about.

When we did this, Transit Gateway was just within the same region. At re:invent they announce the cross region transit gateway which we will look at moving to as well, but for now, it is just a full mesh of VPC peers, which for 8 regions isn't bad... but certainly gets worse with each new region we need to add.

For exposing the service into other VPCs in the same region we use private-link endpoints as to avoid needing to do even more peering.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#167

Post by tylertreat »

alfalfasprout wrote: post #161

I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry. Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support comp…

ZeroMQ is not a message queue, it's a networking library.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#168

Post by james-mcelwain »

nvarsj wrote: post #103
bsaul wrote: post #18

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…

It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…

> the JVM's awful GC

This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful.

Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#170

Post by deanCommie »

jjeaff wrote: post #112
3fe9a03ccd14ca5 wrote: post #108

Earlier quoted context omitted.

The first question I have is why? SQS seems like such a simple thing to keep hosted.

They said they are currently doing 50k messages a second and they aren't even done migrating everything over. 50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check). Plus, with sqs, you get what they have. No customizations.

I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput.

No way they would actually hit that sustained throughput for the entire month.

Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.

Post reply on HN