How is it compared to kafka?
Some latency benchmarks: https://kafkaesque.io/performance-comparison-between-apache-...
Apache Pulsar is an open-source distributed pub-sub messaging system
21–30 of 249 posts
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#22How does this compare with NATS?
NATS is a simpler PUB/SUB system that delivers in the UNIX spirit of small composable parts. Apache Pulsar or Apache Kafka deliver the banana, the ape holding it and the rest of the jungle.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#23Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead.
Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet.
CS is damned, and this is hell.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#24Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
> is pulsar going to take the same road ? Yes, it's in the works
Related Kafka KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...
It doesn't look like it's going to be ready anytime soon though.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#25Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#26This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations?
For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains how Join relates to Set operations:
https://blog.jooq.org/2015/10/06/you-probably-dont-use-sql-i...
Another thing I'd like is if KV stores exposed a general purpose functional programming language (maybe a LISP or a minimal stack-based language like PostScript) for running the same SQL Set operations without ugly syntax. I don't know the exact name for this. But if we had that, then we could build our own distributed databases, similar to Firebase but with a SQL interface as well, from KV stores like Pulsar. I'm thinking something similar to RethinkDB but with a more distinct/open separation of layers.
The hard part would be around transactions and row locking. A slightly related question is if anyone has ever made a lock-free KV store with Set operations using something like atomic compare-and-swap (CAS) operations. There might be a way to leave requests "open" until the CAS has been fully committed. Not sure if this applies to ledger/log based databases since the transaction might already be deterministic as long as the servers have exact copies of the same query log.
Edit: I wrote this thinking of something like Redis, but maybe Pulsar is only the message component and not a store. So the layering might look like: [Pulsar][KV Store (like Redis)][minimal Set operations][SQL query engine].
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#27How does this compare to Redis Pub-Sub or RabbitMQ?
Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.
What about ZeroMQ?
Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn’t it far more performant and distributed?
Maybe I am missing something here.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#28Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead. Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet. CS is damned, and this is hell.
Please do elaborate
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#29This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…
e.g. how about a complex event processing engine? Something like that will do a lot of the above, but the inference database stays managable since old data will fall out of the windows.
Re: Apache Pulsar is an open-source distributed pub-sub messaging system
#30Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
Zookeeper sux. If you are in the Java world you can often roll something better out using Hazelcast.