Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

181–190 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#181
post #112

Earlier quoted context omitted.

They said they are currently doing 50k messages a second and they aren't even done migrating everything over. 50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check). Plus, with sqs, you get what they have. No customizations.

I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput. No way they would actually hit that sustained throughput for the entire month. Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.

Why is this hard to believe?

Maybe they have chatty IoT devices. Maybe a ton of sensors monitoring manufacturing plants for multiple different metrics in real-time. Maybe they just have large scale.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#182

A lot is said or referenced in this conversation about why people chose Pulsar over Kafka. I'm not an expert in this area but are there use cases where Kafka is still better?

As someone with a few years of Kafka and the ecosystem under my belt, but no experience of using Pulsar in anger, the areas I can see where Pulsar is behind are mainly ancillary, and will likely be caught up by the community given a year or two.

Kafka Streaming - Pulsar functions don't intend to (by the looks of it) provide all of the functionality available in Kafka Streaming. E.g., joining streams, treating a stream as a table (a log with a key can be treated as a table), aggregating over a window. They seem to be more focused on do X or Y to a given record. That said, you don't need Kafka Streaming for that, other streaming products like Spark Streaming can do it also (although last I checked, Spark Structured Streaming still had some limitations compared to Kafka Streaming - can't do multiple aggregations on a stream etc.) A use case I have and love Kafka Streaming's KTables for is enriching/denormalising transaction records on their way through the pipeline.

Kafka Connect - Pulsar IO will get there with time, but currently KC has a lot more connectors - for example, Pulsar IO's Debezium source is nice, (Debezium was built to use Kafka Connect, but can run embedded), but you may not want to publish an entire change event stream onto a topic, you might just want a view of a given database table available - so KC's JDBC connector is a lot more flexible in that regard, and Pulsar IO currently doesn't have a JDBC source. It also looks like its JDBC sink only supports MySQL and SQLite (according to the docs) - KC's JDBC connector as a sink has a wider range of support for DBs, and can take advantage of functionality like Postgres 9.5+s UPSERT. Likewise, there's no S3 sources or sinks - the tiered storage Pulsar offers is really nice, but you may only want to persist a particular topic to S3.

KSQL - KSQL lets your BAs etc. write SQL queries for streams. That said, I do like Pulsar SQL's ability to query your stored log data. When I've needed to do this with Kafka, I've had to consume the data with a Spark job, which adds overhead for troubleshooting.

So yeah, that's the main areas I can see, but it's really a function of time until Pulsar or community code develops similar features.

The only other major difference I can see is that at the current time, it's comprised of three distributed systems (Pulsar brokers, Zookeeper, Bookkeeper) which is one more distributed system to maintain with all the fun that entails.

That said, I'll be keeping my eye on this, and trialling it when I get some spare time, as I've found that people will inevitably use Kafka like a messaging queue, and that is a bit clunky. Plus I'm a little over having people ask me how many partitions they need :D

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#183

Earlier quoted context omitted.

Was NATS a consideration for your use cases? At work, we are currently standardizing on NATS as our messaging system, and I would like to know if there is a valid comparison.

Nats is not a replacement for pulsar or rabbitmq. It is a message passing system designed to pass lots of messages live, however if nobody is their to receive them they are lost and gone forever. There is a streaming layer but that is closer to Kafka and still does not provide the typical message model with an ack/nack API. I have used nats in several different ways but since it can be lossy its never been considered…

Thank you for the observation.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#184
post #103

Earlier quoted context omitted.

It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…

> the JVM's awful GC This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful. Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.

> The problem is that most JVM apps just produce too much garbage,

This is strange way of saying that Java the language and most frameworks around it force apps to generate this much garbage.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#185

Earlier quoted context omitted.

Was NATS a consideration for your use cases? At work, we are currently standardizing on NATS as our messaging system, and I would like to know if there is a valid comparison.

I think the other reply captures most of it for core NATS, but we also looked at NATS Streaming a bit, but it seems to be pretty immature (though promising) and doesn't check all the boxes around integrations into the streaming ecosystem like Pulsar does (Pulsar functions, Pulsar IO). I am interested to see where NATS goes but for where we are today Pulsar was a much more obvious choice.

Thank you. I should have imagined it had been asked already.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#186
post #180

Earlier quoted context omitted.

What do you mean by 1 and 2? I'm guessing you're referring to the kafka-clients API? The defaults for producer and consumer conf are quite sensible these days.

I wasn’t around to make those decisions at my company, but I imagine that the “these days” component was the cause? There are a lot of configurations, new ones appear and old ones disappear or change names, etc. In this churny environment, where you want to keep on latest versions (necessitated by bugs mentioned in), you need abstractions to protect you somewhat from the churn. Confluent also seems to have a fair amo…

Sorry, when I say these days, I mean >= Kafka 1.0. Things like auto commit offset in 0.8 days were something like 1 minute, as opposed to 5 seconds onwards, max fetch bytes was set significantly higher etc.

My biggest problems with it were when developers who didn't really understand Kafka started setting properties that had promising names to bad values to "ensure throughput" - let's set max.poll.records to 1 to ensure we always get a record as soon as one is available!

That might be my biggest issue with Kafka - it requires a decent amount of knowledge of Kafka to use it well as a developer. I'm not sure if Pulsar removes that cognitive burden for devs or not, but I'm interested in finding out.

And yeah, the wrappers to remove that burden were written in our company too - but then proved quite limiting for the varying use cases for a Kafka client in our system. sigh

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#187
post #145
post #103

Earlier quoted context omitted.

It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…

For what its worth I have a java microservice running on 13mb of ram.

Without knowing what that service does, how many users/request it serves per second/day, I can only assume it just an http listener up on some port that returns "Hello {username}" when someone sends a GET request.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#188

Earlier quoted context omitted.

The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more flexible than any hosted options. For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard…

> able to have topics be so cheap For GDPR a lot of us has to do exportable 'user activity'. Can you in theory have a topic/user ( we had like 50 million users) and publish any user activity to that topic?

Pulsar docs indicate "millions" of topics but IDK what 50 million would look like but from what I know I would be a bit nervous about it :)

It might be worth chatting with Pulsar devs on their slack community (https://apache-pulsar.herokuapp.com/).

Most commonly what I hear people doing for this is either one of two approaches (or a combination of both): - encrypt the user data and delete the key, eventually the user data will get removed - regularly compact the topic (pulsar has a compaction feature) and write in a tombstone record which will remove any user data after compaction

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#189
post #184

Earlier quoted context omitted.

> the JVM's awful GC This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful. Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.

> The problem is that most JVM apps just produce too much garbage, This is strange way of saying that Java the language and most frameworks around it force apps to generate this much garbage.

No, it's a way of saying that Java != the JVM. The JVM's GC is not "awful" precisely because so many Java frameworks are awful.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#190
post #149

Earlier quoted context omitted.

Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?

That's correct, we're moving to have a pluggable metadata store and coordination service.

Note that this is a different goal from the KIP referenced above, which is to entirely remove any dependency on an external configuration service. The idea of “pluggable consensus” is explicitly rejected in this KIP.
Post reply on HN