Is there a reason you went with Pulsar over Kafka? How is the pulsar community? Where are you turning when you have support issues?
Apache Pulsar is an open-source distributed pub-sub messaging system
151–160 of 249 posts
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#152Most of the comments are just pro-Pulsar but what's the architectural trade-off? (Non-architectural trade-off is that Pulsar is a new system to learn for folks familiar with maintaining and using Kafka.)
Pulsar is better designed than Kafka in every way with the main trade-off being more moving pieces. That's why the recommended deployment is Kubernetes which can manage all that complexity for you.
Pulsar also lacks in the size of the community and ecosystem where Kafka has much more available.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#153I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
We are developing a social app with features such as messaging, notifications etc. We decided to use Yedis [0] (Yugabyte Redis) which is a distributed Redis with persistence backed by RocksDB. Yugabyte supports multiple datacentre distribution. Yedis's pub/sub is distributed as well. We are already running a Yugabyte cluster for data storage in Cassandra. So we didn't have to do anything extra to get our distributed…
Normally don't plug my own work, but this is super related. Did you ever check out Stream? https://getstream.io/
We power chat and feeds for >500 million end users. Tech is Go, RocksDB & Raft based.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#154Earlier quoted context omitted.
Or Firebase?
Or Kafka ?
Read the other comment thread: https://news.ycombinator.com/item?id=21936523
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#155Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…
Nowadays Java based micro-services can compile to native thanks to Graal native-image and frameworks like: https://quarkus.io/.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#156Earlier quoted context omitted.
We are developing a social app with features such as messaging, notifications etc. We decided to use Yedis [0] (Yugabyte Redis) which is a distributed Redis with persistence backed by RocksDB. Yugabyte supports multiple datacentre distribution. Yedis's pub/sub is distributed as well. We are already running a Yugabyte cluster for data storage in Cassandra. So we didn't have to do anything extra to get our distributed…
Normally don't plug my own work, but this is super related. Did you ever check out Stream? https://getstream.io/ We power chat and feeds for >500 million end users. Tech is Go, RocksDB & Raft based.
Yes, we did indeed consider Stream, but figured we could save some money by deploying and running our own system. We are very hopeful to quickly get a couple million users in a short time from our launch and that would have ran up our costs with stream quickly.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#157A lot is said or referenced in this conversation about why people chose Pulsar over Kafka. I'm not an expert in this area but are there use cases where Kafka is still better?
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#158I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
We are developing a social app with features such as messaging, notifications etc. We decided to use Yedis [0] (Yugabyte Redis) which is a distributed Redis with persistence backed by RocksDB. Yugabyte supports multiple datacentre distribution. Yedis's pub/sub is distributed as well. We are already running a Yugabyte cluster for data storage in Cassandra. So we didn't have to do anything extra to get our distributed…
I am not familiar enough with either Yedis or your use case to make a recommendation, but I can say that Pulsar has a great set of features, particularly if you need long term retention, that make it very attractive. I also been impressed with the community and development pace.
Being that the project is a top level Apache project and also has some adoption by quite a few different companies and a number of corporate sponsors the future of the project is pretty safe bet.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#159I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
I have no idea what "pub-sub" is used for outside of its academic definitions, and I have no idea how Hashicorp Vault works - Don't you need a secret/password in cleartext at some point, for a given service or definition? You don't have to answer my questions, I am just shouting into the void. I'm glad it works for y'all.
Pub/sub messaging is super common.
Most IOT devices that aren’t running HTTP stacks are using MQTT.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#160This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receive…
That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements, retries, local storage, idempotency, etc. MSMQ handles that stuff behind the scenes but the problem itself will always exist if there's a network boundary.
These other systems are designed to be remote with a network interface. You can use the client drivers to handle acknowledgements/retries/local-buffering in your own app or use something like Logstash [1], FluentD [2], or Vector [3] for message forwarding if you want a local agent to send to. You might have to wire up several connectors since none of them forward directly to Pulsar today.
Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar.