Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

31–40 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#31

Post by manigandham »

zackmorris wrote: post #26

This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…

Spark [1], Presto [2], and Drill [3] can all do that with connectors to different data sources and varying support for advanced SQL.

Pulsar has support for Presto: https://pulsar.apache.org/docs/en/sql-overview/

Pulsar isn't a KV store though, it's a distributed log/messaging system that supports a "key" for each message that can be used when scanning or compacting a stream. GET and SET aren't individual operations but rather scans through a stream or publishing a new message.

If you just want to have a SQL interface to KV stores or messaging systems that support a message key then Apache Calcite [4] can be used as a query parser and planner. There are examples of it being used for Kafka [5].

1. https://spark.apache.org/

2. https://prestodb.io/

3. https://drill.apache.org/

4. https://calcite.apache.org/

5. https://github.com/rayokota/kareldb

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#32

Post by _frkl »

JackRabbitSlim wrote: post #23

Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead. Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet. CS is damned, and this is hell.

Care to elaborate? I just started using the standalone version of pulsar for a project, it looked better designed than Kafka, resources usage looks quite acceptable so far, but i dont have much experience with any solutions in this space, so im not sure which problems im going to run into. Any suggestions for good tech/strategies for a streaming-type setup like this? Or what to do alternatively? Should i look into rolling my own?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#34

Post by TuringNYC »

PaulHoule wrote: post #29
zackmorris wrote: post #26

This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…

e.g. how about a complex event processing engine? Something like that will do a lot of the above, but the inference database stays managable since old data will fall out of the windows.

I usually dont associated CEP with this, but is makes sense. Are they meant to operate at this level (rather than at a higher level of abstraction)?

Which ones would you recommend looking at?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#35

Post by progval »

SkyRocknRoll wrote: post #16
barbarbar wrote: post #14

How is it compared to kafka?

Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-wit...

> when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads

I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#36

Post by coryvirokmobile »

zackmorris wrote: post #26

This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…

Regarding the generic sql engine - it looks like this is what Apache Calcite was designed for.

https://calcite.apache.org/

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#37

Post by Joeri »

eerrt wrote: post #2

How does this compare to Redis Pub-Sub or RabbitMQ?

It’s closer to redis streams, except like kafka you can scale topics beyond the limits of a single server because they can be distributed. You couldn’t run the twitter firehose over redis streams, but you can run it over pulsar or kafka, given enough hardware.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#38

Post by geodel »

bsaul wrote: post #18

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…

This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java.

Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#40

Post by TuringNYC »

sz4kerto wrote: post #9
eerrt wrote: post #2

How does this compare to Redis Pub-Sub or RabbitMQ?

Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.

I went to https://pulsar.apache.org but didnt find a "Why Pulsar and not Kafka" -- is there an answer to that, or is this another Kafka competitor with the same strengths and not a specific differentiator?
Post reply on HN