How is it compared to kafka?
You might want to check out this blog post I wrote comparing Kafka to Pulsar: https://kafkaesque.io/5-more-reasons-to-choose-apache-pulsar...
Apache Pulsar is an open-source distributed pub-sub messaging system
41–50 of 249 posts
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#42Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead. Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet. CS is damned, and this is hell.
This is one of the best designed pub/sub messaging systems available, but you don't have to use it if you don't want to.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#43How is it compared to kafka?
Separates storage from brokers for better scaling and performance. Millions of topics without a problem and built-in tenant/namespace/topic hierarchy. Kubernetes-native. Per-message acknowledgement instead of just an offset. Ephemeral pub/sub or persistent data. Built-in functions/lambda platform. Long-term/tiered storage into S3/object storage. Geo-replication across clusters.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#44Earlier quoted context omitted.
Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.
What about ZeroMQ? Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn’t it far more performant and distributed? Maybe I am missing something here.
Message queues and message logs do different things. The idea of the log is that subscribers can show up after the log is written and read or reread it from the beginning. In an event sourced architecture you use the log as the source of truth and all consumers can replay the log against a local store to reconstruct a view of the system’s state. You also can use a log for pubsub, but if that’s all you need one of the MQ solutions is usually a better fit.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#45How is it compared to kafka?
You might want to check out this blog post I wrote comparing Kafka to Pulsar: https://kafkaesque.io/5-more-reasons-to-choose-apache-pulsar...
If you have an O'Reilly subscription, you can also check out this detailed report comparing Pulsar and Kafka: https://learning.oreilly.com/library/view/apache-pulsar-vers...
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#46Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java. Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.
NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are much more straightforward.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#47Earlier quoted context omitted.
Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.
I went to https://pulsar.apache.org but didnt find a "Why Pulsar and not Kafka" -- is there an answer to that, or is this another Kafka competitor with the same strengths and not a specific differentiator?
Here is a two-part blog post I wrote on why Pulsar and not Kafka: https://kafkaesque.io/5-more-reasons-to-choose-apache-pulsar...
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#48Earlier quoted context omitted.
Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-wit...
> when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency
Kafka works best when the data it is returning to consumers is in the page cache.
When consumers fall behind, they start to request data that might not be in the page cache, causing things to slow down.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#49Earlier quoted context omitted.
Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-wit...
> when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency
Kafka brokers handle connections to consumers and data storage. This creates contention as the primaries for each partition have to service the traffic and handle IO. Consumers that aren't tailing the stream will cause slowdowns because Kafka has to seek to that offset from files which aren't cached in RAM.
Pulsar separates storage into a different layer (powered by Apache Bookkeeper) which allows consumers to read directly from multiple nodes. There's much more IO throughput available to handle consumers picking up anywhere in the stream.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#50How does this compare with NATS?
NATS is ephemeral pub/sub only. There is no persistence or replay, but focuses on high performance and messaging patterns like request/reply.
Kafka and Pulsar persist every message and different consumers can replay the stream from their own positions. Pulsar also supports ephemeral pub/sub like NATS with a lot more advanced features.
NATS does have the NATS Streaming project for persistence and replay but it has scalability issues. They're working on a new project called Jetstream to replace this in the future.