Being able to scale the durable-storage layer independently has a lot of advantages. More thoughts here: https://twitter.com/breckcs/status/1203736751681896449.
Apache Pulsar is an open-source distributed pub-sub messaging system
141–150 of 249 posts
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#142I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
On behalf of everyone here, thanks a lot for answering every single question being asked. Highly appreciate it. I have questions myself: 1. Did it reduce (TCO) costs or increase it versus using Kinesis and SQS/SNS? 1a. Interestingly, there's no global-replication with those AWS services. Why did you require global-replication with the move to Apache Pulsar? 2. Since you mention internal auth : Weren't Cognito / KMS /…
1. On a short time horizon, not as sure, back of the napkin, it took ~12 dev months (5 months with 2.5 people average on it). However, our cost per 1000 msgs/sec is much lower (like 1/4 the cost of Kinesis) so we fully expect that investment to pay off over time assuming that adoption by the rest of the org continues and we don't find a ton of issues.
1a. You are correct we didn't require geo-replication for existing use cases, however, initially, we saw geo-replication as an easy way to improve DR and we have an internal requirement for a DR zone in another region. Now that we have done the work, we are starting to see multiple places where we can simplify some things with geo-replication, so we think long term the feature will be really valuable
2. We split up auth into two main components: auth of users (where we use Okta) and auth of services. For okta, we just wrote a small webapp that users can log into via OKta and generate credentials. For apps/services, we already had hashicorp in place and wanted to just piggyback of our existing form of identity (IAM roles). Essentially, a user just associates an IAM role with a pulsar role and we generate and drop off credentials into a per-role unique shared location in vault that any IAM role can access (across multiple AWS accounts)
3. Once again, geo-replication wasn't really a hard requirement initially but more of something that we really like now that we have. I think the biggest reason why not postgres is that we have combined message rates (not everything is migrated yet) on the order of 300k msgs/sec across a few dozen services. Pulsar is designed to scale horizontally and also has really great organizational primitives as well as an ecosystem of tools. While I think you could maybe figure that out with some PG solution, having something purpose built really can pay big dividends for when you are trying to make a solution that can easily integrate into a complex ecosystems of many teams and many different apps/use cases
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#143I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
I have no idea what "pub-sub" is used for outside of its academic definitions, and I have no idea how Hashicorp Vault works - Don't you need a secret/password in cleartext at some point, for a given service or definition?
You don't have to answer my questions, I am just shouting into the void. I'm glad it works for y'all.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#144Most of the comments are just pro-Pulsar but what's the architectural trade-off? (Non-architectural trade-off is that Pulsar is a new system to learn for folks familiar with maintaining and using Kafka.)
Someone linked some benchmarks here (on mobile and can’t find) that showed a single node Kafka outperforms but as soon as you start scaling Pulsar pulls ahead. I’m not familiar enough with the nitty gritty to comment beyond that.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#145Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…
For what its worth I have a java microservice running on 13mb of ram.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#146Earlier quoted context omitted.
> Please could you try and explain how you came to this conclusion? 1. Stateless brokers With Kafka any time a broker goes down you need to be aware of the kafka broker id. Yes, this can be fixed by creating your entire infrastructure as code and keeping track of state. This is something of great OpEx. I've seen few people successfully automate this, Netflix is one of the few. The rest just use manual process with to…
I think what is dead is confluent cloud b/c Amazon MSK and Azure HDInsight will be close to feature parity at much less cost.
Damn, I got lazy on my reply & just hoped nobody went further, but well played on digging deeper.
5. Kafka is silly expensive
Pulsar supports message ack with subscription groups. The worst case with Pulsar is you're storing the entire retention period.
Let's say you have a 4 day retention window, to cover an outage happening on Friday and not having to deal with it until Monday. This is pretty typical with what I see in the Kafka world for small-mid size companies who don't want to pay the 1.5x OT on call.
So, with Pulsar you're at worst storing the 4 days of data but at best you're only storing the messages within the lag period of all consumer groups acknowledging the message.
Now, without getting too deep into Pulsar's feature set even that is a lie because Pulsar has tiered storage as a first class citizen. The messages after the four days could be ship off to S3 if we wanted or even within 1 day depending on our use case and this is all built into Pulsar, no OpEx tooling required. Even access the messages from S3 through Pulsar is abstracted, there's no tooling required to pull them back in if you wanted.
Now with Kakfa our worst case is simply 4 days of retention data. This can get very expensive as compute & storage are tied together, it means scaling up all the brokers (even though we don't need the throughput) for the storage increase. Now, yes MSK basically abstracts all this from you but you're paying for it.
6. AWS Managed Service are not equal citizens to EC2 standalone
Managed services right now don't fall under the new Saving Plan: https://aws.amazon.com/blogs/aws/new-savings-plans-for-aws-c...
This will cost you 30-60% discount on your entire Kafka bill.
7. Excel Life
If I look at the numbers for what I'm doing it would have costed ~$4M for Kafka vs ~$1M for Pulsar.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#147Earlier quoted context omitted.
Why did you choose Pulsar?
The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more flexible than any hosted options. For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard…
> able to have topics be so cheap
For GDPR a lot of us has to do exportable 'user activity'. Can you in theory have a topic/user ( we had like 50 million users) and publish any user activity to that topic?
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#148I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
We are developing a social app with features such as messaging, notifications etc. We decided to use Yedis [0] (Yugabyte Redis) which is a distributed Redis with persistence backed by RocksDB. Yugabyte supports multiple datacentre distribution. Yedis's pub/sub is distributed as well. We are already running a Yugabyte cluster for data storage in Cassandra. So we didn't have to do anything extra to get our distributed pub/sub up and running. Would you recommend using Pulsar instead?
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#149Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
> is pulsar going to take the same road ? Yes, it's in the works
Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#150Earlier quoted context omitted.
The first question I have is why? SQS seems like such a simple thing to keep hosted.
Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cases, it is being able to have one tech that does SQS like stuff and Kinesis like stuff in one place
When we ran into a similar use case we went with writing to S3 and using S3 SNS notifications and consumers could subscribe their SQS queues to that.
Having said that, assuming Pulsar has a similar feature to Kafkas log compaction I can definitely see the appeal!
Also the fact that SQS Fifo doesn’t integrate with this setup is super annoying.
Edit: Log compaction not key compaction