Can pulsar users here talk about exactly once message delivery from the consumer side? I use Kafka and needed a cache based on incoming events. But if the consumer crashes it's not easy to pick up from exactly where it left off without manually committing offsets which hurts performance. There's also some hand waving Kafka gossip that it's hard to commit offsets right. Any insight here is greatly appreciated.
Pulsar vs. Kafka
91–100 of 103 posts
Re: Pulsar vs. Kafka
#92Earlier quoted context omitted.
What kind of (lower level) surrogate metrics would you be interested in that could translate to '3am phone calls' when comparing messaging systems?
Being used by at least one company of significant size that (a) i've heard of and (b) isn't directly connected to the project would be a good start.
'Directly connected' to a project might mean a user of - but I assume you mean a major contributor to (as even small time users of free software often contribute something - bug reports, feature requests, code contributions, money, etc).
The page: https://pulsar.apache.org/powered-by/ suggests there's quite some number of corporate users who are happy to confirm they use this suite. I don't know how many of those you've heard of, though.
I suspect many private & government agencies around the world would decline to formally attach their name to any list like this, lest it be (mis)interpreted as an endorsement.
Re: Pulsar vs. Kafka
#93Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).
On the issue of delivery ... can Pulsar users here talk about exactly once message delivery from the consumer side?
I use Kafka and needed a cache based on incoming events which were partitioned.
But if the consumer crashes it's not easy to pick up from exactly where it left off without manually committing offsets which hurts performance. There's also some hand waving Kafka gossip that it's hard to commit offsets right. Further suppose a task consumes events from topic A partition 3, produces events to topic B. Again, if the task crashes it's not clear what needs replaying in order to not to lose messages (on write) or missing messages (on read).
Any insight here is greatly appreciated.
Re: Pulsar vs. Kafka
#94Another service that "needs" zookeeper. Has anyone figured out a simple way to manage zookeeper for a small team?
I keep hearing this, and I've had this impression myself. But I've ben running a small ZK ensemble for two years just to support a Kafka cluster. Between them, ZooKeeper has been much less trouble, almost zero problems.
Re: Pulsar vs. Kafka
#95AWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems). It's not all hype either, according to this post https…
Instead of Kinesis we you AWS MSK (Managed Kafka), which is expensive but works quite well.
Calling MSK expensive is maybe not being clear enough. It's more than 2x the cost of the raw EC2 instances. Then on top of that, to get metrics at the per broker level there's an upcharge, and a further one for per topic metrics. These are things that are actually extremely important at scale, and it's absurd how expensive it gets.
Then you combine that with how immature the UX is, and it really just doesn't feel good to deal with. It's not that much work to run a Kafka cluster with the sort of design that MSK provides, and it can be done better than that without much cost.
Re: Pulsar vs. Kafka
#96Can anyone share their thoughts on whether, in case of a new project is it worth to start with Pulsar instead of Kafka as a distributed log/pub sub solution with guaranteed delivery? I heard a lot of stories about Kafka's operational complexity and TFA seems to be pointing out that Pulsar has a lower operational upkeep (ie. less manpower needed to keep it running).
On the issue of delivery ... can Pulsar users here talk about exactly once message delivery from the consumer side? I use Kafka and needed a cache based on incoming events which were partitioned. But if the consumer crashes it's not easy to pick up from exactly where it left off without manually committing offsets which hurts performance. There's also some hand waving Kafka gossip that it's hard to commit offsets rig…
Unlike Kafka, and despite some unfortunately misleading wordings in articles and documentation pages, Pulsar doesn't actually support exactly-once aka 'effectively-once' semantics because it lacks support for transactions. It only supports an idempotent producer combined with message deduplication. The current functionality only works when producing one message and to only one partition. For example, you cannot atomically produce multiple messages to one partition with Pulsar today, let alone multiple partitions.
See this Dec 2019 presentation by Pulsar committers, where they explain all this in more detail, i.e., the lack of transactions and the resulting limitations, and the motivation for adding such transactions to Pulsar. The approach looks very similar to Kafka's. https://www.slideshare.net/streamnative/transaction-preview-... The original ETA for transactions was Pulsar v2.6 (June 2020), but as of today there's still quite some work to be done (https://github.com/apache/pulsar/issues/2664). The latest ETA seems to be around the end of the year.
The key difference for an end user is that Kafka released all the functionality in one go back in 2017 (idempotent producer, transactions; which fwiw also explains why designing+building+testing took the Kafka community that long) so it has been much easier to understand what is actually supported vs. what is not.
Re: Pulsar vs. Kafka
#97Earlier quoted context omitted.
That's true but still limitation is not fully resolved. In order to increase consumption rate, we need to add replicas. In pulsar Brokers are merely cache nodes over Bookkeeper. Adding more Brokers is trivial in Pulsar.
How in pulsar do they get around the fact that adding a new broker, data needs to be moved over before that broker can start serving data? This seems like a basic law of physics type limitation to me.
(copying this text from another comment of mine elsewhere)
Well, the Pulsar broker is (kinda) stateless, because they are essentially a caching layer in front of BookKeeper. But where's your data actually stored then? In BookKeeper bookies, which are stateful. Killing and replacing/restarting a Bookkeeper node requires the same redistribution of data as required in Kafka’s case. (Additionally, BookKeeper needs a separate data recovery daemon to be run and operated, https://bookkeeper.apache.org/archives/docs/r4.4.0/bookieRec...)
So the comparison of 'Pulsar broker' vs. 'Kafka broker' is very misleading because, despite identical names, the respective brokers provide very different functionality. It's an apples-to-oranges comparison, like if you'd compare memcached (Pulsar broker) vs. Postgres (Kafka broker).
Re: Pulsar vs. Kafka
#98Earlier quoted context omitted.
Being used by at least one company of significant size that (a) i've heard of and (b) isn't directly connected to the project would be a good start.
'Directly connected' to a project might mean a user of - but I assume you mean a major contributor to (as even small time users of free software often contribute something - bug reports, feature requests, code contributions, money, etc). The page: https://pulsar.apache.org/powered-by/ suggests there's quite some number of corporate users who are happy to confirm they use this suite. I don't know how many of those you…
I've heard of Comcast, but that's Yahoo. Not heard of the others.
Re: Pulsar vs. Kafka
#99Earlier quoted context omitted.
Nightly to seconds is definitely great. Seconds to milliseconds is what is questionable I think.
> Nightly to seconds is definitely great Why? Can a business mobilise in anything less than days? If a report is minutes out of data, is that any loss? Given that some largish proportion of reports are never used, perhaps better management is key. not disagreeing but efficiency is not just a matter of quickness.
I work in oil and gas with process optimization, and we are working on autonomous tuning of the production process in a way that is not possible with daily data updates and manual human intervention.
Quite simply, the end goal is increased oil production and reduced power consumption. Realtime data flows are a big deal when reading from thousands of sensors for just one process system and there are hundreds of tunable parameters. All this has to be safe also.
Re: Pulsar vs. Kafka
#100Earlier quoted context omitted.
Pulsar is better for very large scale deployments provided you have people to manage it
Kafka is handling very large scale deployments just fine atm in all the big tech co's. The only thing I can see that can make this true is Pulsar seems to have better elastic scalability. But it seems to score less on everything else. It has a much more complex storage system that ends up not matching Kafka's high-end throughput at large scale. From what I recall, Twitter ended up abandoning BookKeeper due to storage…
This is mostly due to the difficulties scaling DistributedLog more so than BookKeeper. DistributedLog basically had no contributors other than Twitter and was just too big of a mountain to climb alone. The blog post you linked goes somewhat into this but that is ultimately why the choice to transition away was made.
Pulsar likely would have been considered if it was more mature at the time and sported a community of comparable size to Kafka (it's still a long way from this).