Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

81–90 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#81

Post by _frkl »

geodel wrote: post #74
_frkl wrote: post #56

Earlier quoted context omitted.

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

Did you check this? https://docs.nats.io/nats-streaming-concepts/intro "..Message/event persistence - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations." and "At-least-once-delivery - NATS Streaming offers message acknowledgements between publisher and server (for…

No, i missed that. I think ive seen 'nats streaming', but didn't realize that it is its own distinct thing. All this makes more sense now to me, thanks!

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#82

Post by bubbleRefuge »

addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

How does python stream processing work. Are the modules running in the JVM ? Jython ?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#83

Post by cbartholomew »

mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

We provide a SaaS offering of Apache Pulsar in AWS, Azure, and GCP: https://kafkaesque.io/

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#84

Post by matteomerli »

bubbleRefuge wrote: post #82
addisonj wrote: post #68

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

How does python stream processing work. Are the modules running in the JVM ? Jython ?

The user code it's all running in a native CPython interpreter.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#85

Post by eronwright »

PaulHoule wrote: post #29
zackmorris wrote: post #26

This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…

e.g. how about a complex event processing engine? Something like that will do a lot of the above, but the inference database stays managable since old data will fall out of the windows.

Take a look at the Apache Flink CEP library, which operates over unbounded streams: https://ci.apache.org/projects/flink/flink-docs-stable/dev/l...

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#86

Post by spelunker »

dickjocke wrote: post #60
sz4kerto wrote: post #9

Earlier quoted context omitted.

Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.

Why is one Apache providing competing with another one?

Plenty competing Apache projects exist, Pulsar and Kafka aren't unique in that regard.

I don't think Apache cares if it's maintaining similar projects.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#87

Post by eronwright »

mindw0rk wrote: post #70
buboard wrote: post #57

Earlier quoted context omitted.

yeah i m talking more generally about their full list here: https://www.apache.org/

There are a lot of projects that were handed to Apache to manage. Kafka for example was initially created by LinkedIn. So yeah, you are right, big corps are actually creating those tools, and in addition to this, giving it away as open source to public.

Best part is that the tools are put into production before being open-sourced. In other words, they actually work.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#88

Post by takeda »

lonesword wrote: post #17
samzer wrote: post #4

For a moment I thought Bajaj and TVS came together.

Captain here. TVS and Bajaj are major motorbike manufacturers in India, and TVS had a model named "Apache" and Bajaj had a model named "Pulsar". Flies away

It's not obvious if you're not from India, so thank you for the explanation :)

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#89

Post by reggieband »

I'm still on the fence with these distributed log/queue hybrids. From a theoretical perspective it seems these are excellent. I just have this nagging suspicion that there is some even-worse problem architectures based on these systems will harbor. This kind of ambivalence is something I find myself having to battle more and more in my career as I age. Most of the time the hype around new design/development patterns leads to a worse situation. Very rarely it leads to a significant improvement. I dislike that my first impression looking at a system like this is risk aversion.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#90

Post by mattboyle »

cbartholomew wrote: post #83
mattboyle wrote: post #62

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt f…

We provide a SaaS offering of Apache Pulsar in AWS, Azure, and GCP: https://kafkaesque.io/

I didnt find this when looking, thanks will take a deeper look.
Post reply on HN