Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

231–240 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#231

Post by injinj »

tylertreat wrote: post #167
alfalfasprout wrote: post #161

I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry. Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support comp…

ZeroMQ is not a message queue, it's a networking library.

I'll bet he is aware. The problem with the trading industry is that they have hundreds of users with bespoke solutions catering to extreme performance criteria rather than hundreds of thousands like NATS. They will keep reinventing the wheel every time a nanosecond can be saved by the latest hardware stack.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#232

Post by perfect_kiss »

ckdarby wrote: post #102
mattboyle wrote: post #91

Earlier quoted context omitted.

It was about 6 months ago. I completely disagree with the opex of picking up kafka vs developing a whole client library. Please could you try and explain how you came to this conclusion?

> Please could you try and explain how you came to this conclusion? 1. Stateless brokers With Kafka any time a broker goes down you need to be aware of the kafka broker id. Yes, this can be fixed by creating your entire infrastructure as code and keeping track of state. This is something of great OpEx. I've seen few people successfully automate this, Netflix is one of the few. The rest just use manual process with to…

While bare metal Kafka does really bundle itself with lots of OpEx trouble, have you ever tried using an orchestrator to manage it ?

DC/OS implementation easily shuns out 1. and 2.

3. and 4. are valid points, but I think in a real life these scenarios are usually related to cloud service cost optimization, and I would never recommend anyone running Kafka in a cloud due to these reasons.

There one more reason, which was not cited, but poses itself a real killer for cloud Kafka dream AFM: clouds, being prone to all kinds of network interruptions, are not well suited for running Zookeeper ensembles with decent uptime.

Disclaimer: I have never tried or used Apache Pulsar, and just examining its documentation after spotting this thread.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#233

Post by namelosw »

bsaul wrote: post #18

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…

It seems to be the trend. There would be more and more equivalents in Go in a foreseeable future. There would be a split.

I generally don't use Java or Go in my applications, but Go components are usually more lightweight and easier to use if you're not majorly working with those runtimes. JVM has a big overhead, and Java applications usually make things worse if they use things like dependency injection.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#234

Post by manigandham »

GordonS wrote: post #226
manigandham wrote: post #160

Earlier quoted context omitted.

That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements, retries, local storage, idempotency, etc. MSMQ handles that stuff behind the scenes but the problem itself will always exist if there's a network boundary. These other systems are designed to be remote with a network interface. You can use the client drivers t…

> Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available. I've never heard of using Redis for pub/sub - wer…

Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc.

It's poor architecture and implementation. One of the worst products built with Erlang. It's hard to work on (for new contributors) while not really using any of the natural advantages of the language.

Redis supports pub/sub channels. It's very fast and if you already use it then it saves running another system.

Streaming log vs pub/sub is a fuzzy distinction and basically has no difference at this point. Publishers send to a topic and consumers listen. You can do the ephemeral pub/sub in Kafka by reducing retention to seconds, using a single partition per pub/sub topic, and having consumers listen from latest without a consumer group. Or use Pulsar which does both much more naturally.

Also I didn't mention but there are great commercial messaging products like AMPS [1] or Solace [2] if you need much more advanced features and support.

1. https://www.crankuptheamps.com/

2. https://solace.com/

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#235

Post by manigandham »

deanCommie wrote: post #170
jjeaff wrote: post #112

Earlier quoted context omitted.

They said they are currently doing 50k messages a second and they aren't even done migrating everything over. 50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check). Plus, with sqs, you get what they have. No customizations.

I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput. No way they would actually hit that sustained throughput for the entire month. Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.

We do 100k msgs/sec minimum in our streaming platform. This is on the low end for adtech and other industries.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#236

Post by manigandham »

perfect_kiss wrote: post #232
ckdarby wrote: post #102

Earlier quoted context omitted.

> Please could you try and explain how you came to this conclusion? 1. Stateless brokers With Kafka any time a broker goes down you need to be aware of the kafka broker id. Yes, this can be fixed by creating your entire infrastructure as code and keeping track of state. This is something of great OpEx. I've seen few people successfully automate this, Netflix is one of the few. The rest just use manual process with to…

While bare metal Kafka does really bundle itself with lots of OpEx trouble, have you ever tried using an orchestrator to manage it ? DC/OS implementation easily shuns out 1. and 2. 3. and 4. are valid points, but I think in a real life these scenarios are usually related to cloud service cost optimization, and I would never recommend anyone running Kafka in a cloud due to these reasons. There one more reason, which w…

> "using an orchestrator to manage it"

These can be just as fragile and now you have to learn how to manage the orchestrator. Even Confluent's own Kubernetes operator has issues. There's just too many issues with Kafka's design that hinders easy operations.

> "I would never recommend anyone running Kafka in a cloud"

That's a major problem considering that's where most computing is heading. At this point, running in noisey overloaded cloud environments is a good test of the reliability and durability of a software system. Kafka fails massively here.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#237

Post by manigandham »

jjjensen90 wrote: post #95
vorpalhex wrote: post #61

Earlier quoted context omitted.

NATS is amazing but note that it makes different promises than Pulsar. NATS doesn't offer true durability (in exchange for amazing performance and great simplicity) whereas Pulsar and similar are meant to survive certain partition or failure situations and not lose data. It's not one or the other, they're just different tools.

There is nats-streaming-server as well which offers true durability (via file or SQL store) and a streaming model very similar to Kafka and Pulsar. It can also run as a raft cluster or in fault tolerance mode. It still has very good performance and is very simple to deploy and operate (I use it for event sourcing for real time IoT data at my day job).

NATS Streaming has major scalability problems even if it's simple to deploy. It's only high-availability unless you the Raft clustering but that has been bolted on to the original project and isn't really well-designed.

The team is working on an entirely new system called Jetstream to eventually replace it.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#239

Post by nostrebored »

ryeguy wrote: post #202
nostrebored wrote: post #179

Earlier quoted context omitted.

Is there any reason you've decided not to use Dynamo to manage your materialized views? Do you need to be able to generate a materialized view for a specific time window? It feels weird to me to use your pub sub system to handle your persistent storage for views, but I am definitely missing context into pulsar and your use case

Pulsar wouldn't be the thing that stores the view, it would be the canonical data store used to rebuild the views. The idea is you have an event sourced view and can rebuilt it at any time by resetting the stream pointer to the beginning.

Makes sense, so in this case realistically it would be replacing the data lake (likely S3) which currently in an AWS event driven architecture requires a lambda function to push to S3 and is vulnerable to network issues.

Seems like a cool use case!

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#240

Post by GordonS »

manigandham wrote: post #234
GordonS wrote: post #226

Earlier quoted context omitted.

> Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available. I've never heard of using Redis for pub/sub - wer…

Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc. It's po…

I didn't k ow about Redis pub/sub, I'll check it out! With a quick glance, I'm not sure it supports message durability (persisting messages to storage)?

IME, any Erlang system is difficult for noobs to contribute to; Erlang's syntax alone is... frightening :)

I really haven't hit any of the issues you mentioned with rabbit. Several clusters in production for years have been rock solid, connections are stable (and when they do drop, the client libraries, of which there are a multitude, can handle reconnection), throughput is excellent, and I've never once seen data corruption (and combined we must have processed many billions of messages).

The whole HIPE debacle I can at least agree on though! (I seem to recall it's deprecated now, but I might have dreamed that)

Post reply on HN