Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

131–140 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#131

This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receive…

You're free to have queue and workers run on the same machine, just bind to loopback. As soon as you deal with more than one machine, which is required in HA scenarios, you deal with a networked (distributed) system. I might not have understood your question correctly though ...

Edit: Maybe you're looking for acks/confirms? https://www.rabbitmq.com/confirms.html

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#132

Earlier quoted context omitted.

It was about 6 months ago. I completely disagree with the opex of picking up kafka vs developing a whole client library. Please could you try and explain how you came to this conclusion?

> Please could you try and explain how you came to this conclusion? 1. Stateless brokers With Kafka any time a broker goes down you need to be aware of the kafka broker id. Yes, this can be fixed by creating your entire infrastructure as code and keeping track of state. This is something of great OpEx. I've seen few people successfully automate this, Netflix is one of the few. The rest just use manual process with to…

I think what is dead is confluent cloud b/c Amazon MSK and Azure HDInsight will be close to feature parity at much less cost.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#133

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

What is the SQS-based system you are migrating from? I'm currently building a data processing system that is backed by S3 -> SQS based events, for persistent message passing.

We have a number of systems, some use SQS, some use Kinesis. Part of the draw of Pulsar is having one piece of tech that we can unify everything over and offer more baseline features, like infinite retention via storage offloading or Pulsar IO connectors that standardize common operations. We aren't really targeting one use case, instead, we looked for the system that offered a broad set of features that other developers in the company want and is operationally doable with just a few people.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#135

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

Was NATS a consideration for your use cases? At work, we are currently standardizing on NATS as our messaging system, and I would like to know if there is a valid comparison.

I think the other reply captures most of it for core NATS, but we also looked at NATS Streaming a bit, but it seems to be pretty immature (though promising) and doesn't check all the boxes around integrations into the streaming ecosystem like Pulsar does (Pulsar functions, Pulsar IO).

I am interested to see where NATS goes but for where we are today Pulsar was a much more obvious choice.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#136

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…

On behalf of everyone here, thanks a lot for answering every single question being asked. Highly appreciate it.

I have questions myself:

1. Did it reduce (TCO) costs or increase it versus using Kinesis and SQS/SNS?

1a. Interestingly, there's no global-replication with those AWS services. Why did you require global-replication with the move to Apache Pulsar?

2. Since you mention internal auth: Weren't Cognito / KMS / Secrets Manager up to the job? Given these are integrated out-of-the-box with EC2?

3. Was it ever under-consideration to roll out pub/sub on top of Aurora for Postgres with Global Replication? https://layerci.com/blog/postgres-is-the-answer/

Thanks again.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#137

This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…

One of the challenges with layering SQL on top of a KV store is query performance.

The most obvious way to model a secondary index on top of a pure KV store is to map indexed values to keys. For example, given the (rowID, name) tuples (123, "Bob"), (345, "Jane"), (234, "Zack"), you can store these as keys:

  name:Bob:123
  name:Jane:345
  name:Zack:234
At this point you don't need or even want values, so this is effectively a sorted set.

Now you can easily find the rowID of Jane by doing a key scan for "name:Jane:", which should be efficient in a KV store that supports key range scans. You can do prefix searches this way ("name:Jane" finds all keys starting with "Jane"), as well as ordinal constraints ("age > 32", which requires that the age index is encoded to something like:

  age:Bob:\x00\x00\x00\x20:123
To perform an union ("name = 'Bob' OR name = 'Jane'"), you simply do multiple range scans, performing a merge sort-ish union operation as you go. To perform an intersection ("name = 'Bob' AND age > 10"), you find the starting point for all the terms and use that as the key range, then do the merge sort.

This is what TiDB and FoundationDB's record layers do, which both have a strict separation between the stateless database layer and the stateful KV layer.

The performance bottleneck will be the network layer. Your range scan operations will be streaming a lot of data from the KV store to the SQL layer, and potentially you'll be reading a lot of data that is discarded by higher-level query layers. This is why TiKV has "co-processor" logic in the KV store that knows how to do things like filter; when TiDB plays your query, it pushes some query operators down to TiKV itself for performance.

Unfortunately, this is not possible with FoundationDB. This is why FoundationDB's authors recommend you co-locate FDB with your application on the same machine. But since FDB key ranges are distributed, there's no way to actually bring the query code close to the data (as far as I know!).

I'm sure you could do something similiar with Redis and Lua scripting, i.e. building query operators as Lua scripts that worked on sorted sets. I wouldn't trust Redis as a primary data store, but it can be a fast secondary index.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#138
post #18

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…

I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#139

I'm still on the fence with these distributed log/queue hybrids. From a theoretical perspective it seems these are excellent. I just have this nagging suspicion that there is some even-worse problem architectures based on these systems will harbor. This kind of ambivalence is something I find myself having to battle more and more in my career as I age. Most of the time the hype around new design/development patterns…

Your risk aversion seems justified. It seems reasonable to estimate that very few teams are in the position of needing the kind of scale/scalability that something like Apache Pulsar offers. They are much more likely to be in either a state where they will not put Pulsar through its paces or where they already have a solution in place that serves their scale/scalability needs.

When a team you are on starts discussing switching over to a technology like Pulsar because of its amazing benefits, unless your pants are on fire, it is much more likely than not that you do not stand to gain much from the benefits that such software brings but you are accepting the maintenance burden that it represents.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#140
post #131

This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receive…

You're free to have queue and workers run on the same machine, just bind to loopback. As soon as you deal with more than one machine, which is required in HA scenarios, you deal with a networked (distributed) system. I might not have understood your question correctly though ... Edit: Maybe you're looking for acks/confirms? https://www.rabbitmq.com/confirms.html

I have many machines, each of which have one or many applications that send messages. And I have one machine with an instance of Rabbit to which all messages are sent. If the network is down or the Rabbit machine is down, the messages are gone along with their data.

Clustering the Rabbit machine helps one particular failure scenario, but it's not a solution to the problem.

Post reply on HN