I really enjoyed the talk given by Quentin Adam & Steven Le Roux at Devoxx 2019. It gave a great overview of Apache Pulsar. Hope someone else can find it useful too!
Apache Pulsar is an open-source distributed pub-sub messaging system
201–210 of 249 posts
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#202Earlier quoted context omitted.
Biggest pain points with Kinesis: - ordering is really hard, you don't get guaranted ordering unless you write one message at a time or do a lot of complexity on writes (see https://brandur.org/kinesis-order ) and the shards are simply too small for many of our ordered use cases - cost, we just don't send some data right now because it would just be too much relative to the utility of the data (we would need like 250…
Is there any reason you've decided not to use Dynamo to manage your materialized views? Do you need to be able to generate a materialized view for a specific time window? It feels weird to me to use your pub sub system to handle your persistent storage for views, but I am definitely missing context into pulsar and your use case
Pulsar wouldn't be the thing that stores the view, it would be the canonical data store used to rebuild the views.
The idea is you have an event sourced view and can rebuilt it at any time by resetting the stream pointer to the beginning.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#203Earlier quoted context omitted.
Was NATS a consideration for your use cases? At work, we are currently standardizing on NATS as our messaging system, and I would like to know if there is a valid comparison.
Nats is not a replacement for pulsar or rabbitmq. It is a message passing system designed to pass lots of messages live, however if nobody is their to receive them they are lost and gone forever. There is a streaming layer but that is closer to Kafka and still does not provide the typical message model with an ack/nack API. I have used nats in several different ways but since it can be lossy its never been considered…
Sounds like zeromq to me
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#204Earlier quoted context omitted.
> Why use RabbitMQ and Kafka if you can use ZeroMQ? They are totally different, you're comparing apples with oranges. ZeroMQ gives you basic, very fast tooling to communicate between distributed processes. ZeroMQ does not provide tooling for e.g. maintaining a strictly ordered, multi-terabyte event log. And so on.
Yes but isn’t this a bit like comparing git / bitkeeper vs subversion / perforce? Basically, one is decentralized and you can set up a massively parallel architecture, with eg each topic or subthread having its own pubsub. The other is a monolithic centralized pubsub architecture. You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn…
Zeromq is just a bit of sugar on top of tcp sockets. It isn't a message queue or anything close. You would be wasting a ton of time reimplementing a lot of basic features like retries, persistence, service discovery, dead letter queues, priority, and a ton of other stuff.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#205I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry. Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support comp…
Aeron[1] is something I have been look at as well.
I like that it is adaptable to run without an external broker (using embedded media driver option), yet can add that for scalability/redundancy.
Our use cases do not require a query on top of streams. Instead all data would go into timescaleDB [2].
[1] https://github.com/real-logic/aeron/wiki/Java-Programming-Gu...
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#206I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
We are developing a social app with features such as messaging, notifications etc. We decided to use Yedis [0] (Yugabyte Redis) which is a distributed Redis with persistence backed by RocksDB. Yugabyte supports multiple datacentre distribution. Yedis's pub/sub is distributed as well. We are already running a Yugabyte cluster for data storage in Cassandra. So we didn't have to do anything extra to get our distributed…
Redis pub/sub is simplistic and ephemeral. There's no persistence or really anything beyond sending some data to any listeners who may be active at that moment.
Redis Streams now offers a capable persistent message stream solution and if/when Yugabyte supports that then it might be a good competitor. Otherwise Kafka/Pulsar offer much for persistent streams.
There's also KeyDB which is a forked Redis that comes with multithreading and persistence to support larger-than-RAM workloads and active/active failover. https://github.com/JohnSully/KeyDB
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#207Earlier quoted context omitted.
NATS is a simpler PUB/SUB system that delivers in the UNIX spirit of small composable parts. Apache Pulsar or Apache Kafka deliver the banana, the ape holding it and the rest of the jungle.
Check out Liftbridge ( https://liftbridge.io ) as a way to add these capabilities to NATS.
your FAQ still says it's not production ready? Is this still the case, I've been keeping my eye on this project
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#208I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
I have no idea what "pub-sub" is used for outside of its academic definitions, and I have no idea how Hashicorp Vault works - Don't you need a secret/password in cleartext at some point, for a given service or definition? You don't have to answer my questions, I am just shouting into the void. I'm glad it works for y'all.
Event based systems -> which almost every microservice is nowadays.
Or chatapps
Or for IOT
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#209Earlier quoted context omitted.
They said they are currently doing 50k messages a second and they aren't even done migrating everything over. 50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check). Plus, with sqs, you get what they have. No customizations.
Even if you were off by an order of magnitude, that expenditure level is not justified to run a message broker service.
5k a month would be an amazing deal.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#210Earlier quoted context omitted.
Not questioning your judgement but interested to know about the factors moving you away from Kinesis.
Biggest pain points with Kinesis: - ordering is really hard, you don't get guaranted ordering unless you write one message at a time or do a lot of complexity on writes (see https://brandur.org/kinesis-order ) and the shards are simply too small for many of our ordered use cases - cost, we just don't send some data right now because it would just be too much relative to the utility of the data (we would need like 250…
Thanks for the write up. We are moving away from Kafka to Kinesis, and will consider these points mentioned.