License? It doesn't seem to be open sourced.
More than technology it is cloud service. So I think code is not the most interesting part here.
Ursa: A leaderless, object storage–based alternative to Kafka
11–20 of 42 posts
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#12License? It doesn't seem to be open sourced.
I am one of the co-founders of StreamNative.
Currently Ursa is only available in our cloud service. But we do plan to open-source the core soon. Stay tuned.
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#13Ursa published a blog post saying their leaderless, stateless, object storage–based Kafka replacement can reduce costs by up to 95%. Has anyone here tried Ursa in production? How much cost reduction have you actually seen compared to Kafka or MSK in real workloads?
As near as I can tell, the claims of huge cost savings derive from the difficulty dynamically scaling Kafka and improved multitenancy. So if different pieces of your company each have overprovisioned kafka clusters, they could all move to Ursa and save all the overprovisioning. I have not tried it, and full disclosure, I really like Kafka: it's one of the pieces of software that has been rock solid for me. I built a…
We also love Kafka as a protocol. However, the implementation can be evolved to adopt the current cloud infrastructure and and rethought based on the modern lakehouse paradigm. That was one of the reasons we created Ursa.
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#14Ursa published a blog post saying their leaderless, stateless, object storage–based Kafka replacement can reduce costs by up to 95%. Has anyone here tried Ursa in production? How much cost reduction have you actually seen compared to Kafka or MSK in real workloads?
As near as I can tell, the claims of huge cost savings derive from the difficulty dynamically scaling Kafka and improved multitenancy. So if different pieces of your company each have overprovisioned kafka clusters, they could all move to Ursa and save all the overprovisioning. I have not tried it, and full disclosure, I really like Kafka: it's one of the pieces of software that has been rock solid for me. I built a…
The bulk of the cost savings comes from the use of object storage rather than attached disks. This eliminates the inter-AZ networking costs associated with Kafka replication mechanism.
I break all of the costs down in the following e-book. https://streamnative.io/ebooks/reducing-kafka-costs-with-lea...
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#15Was the key unlock here the ability to append data to an object? ( https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... )
Having built a prototype of a system like Ursa myself, this isn't something that you need to use at all, especially because it seems like this is only available in S3 Express One Zone.
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#16Was the key unlock here the ability to append data to an object? ( https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... )
There are a few things unlocked by Ursa:
1. It is leaderless by design. So there is no single lead broker you need to route the traffic. So you can eliminate majority of the inter-zone traffic.
2. It is lakehouse-native by design. It is not only just use object storage as the storage layer, but also use open table formats for storing data. So streaming data can be made available in open table formats (Iceberg or Delta) after ingestion. One example is the integration with S3 Tables: https://aws.amazon.com/blogs/storage/seamless-streaming-to-a... This would simplify the Kafka-to-Iceberg integration.
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#17To me it seems Pulsar, a stream native sponsored project has not picked up. So a wrapper over Kafka/Pulsar with all Kafka compatibility and perhaps pulsar technology in cloud streaming engine is good business play.
There seems to be a confusion here.
Pulsar has been widely adopted in many mission-critical business-facing systems like billing, payment, transaction processing, or used a unified platform that consolidate enterprises diverse streaming & messaging use cases. It has quite a lot of adoptions from F500 companies, hyperscalers, to startups.
Kafka is used for in data ingestion and streaming pipeline. Kafka protocol itself is great. However, the implementation has its own challenges.
Both Pulsar and Kafka are great open source projects and their protocols are designed for different use cases. We have seen many different companies use both technologies.
Ursa is the underlying streaming engine that we re-implemented to be leaderless and lakehouse-native so that we can better leverage the current cloud infrastructure and natively integrate with broader lakehouse ecosystem. It is the engine we used to support both in our product offerings.
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#18Was the key unlock here the ability to append data to an object? ( https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... )
Having built a prototype of a system like Ursa myself, this isn't something that you need to use at all, especially because it seems like this is only available in S3 Express One Zone.
Ursa is available across all major cloud providers (GCP, Azure, AWS). It also supports pluggable write ahead log storage. For latency relaxed workloads, we use object storage to get the cost down. So it works with AWS S3, GCP GCS, Azure Blob Store. For latency sensitive workloads, we use Apache BookKeeper which is a low-latency replicated log storage. This allows us to support workloads ranging from milliseconds to sub-seconds. You can tune it based on latency and cost requirements.
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#19Congrats on the launch! This is Zhenni from PuppyGraph. Shameless plug - We recently supported Ursa and here is the joint blog to showcase how to integrate Ursa engine with PuppyGraph to enable real-time graph analytics for a financial service use case with data stored in a lake house (not graphDB): https://streamnative.io/blog/integrating-streamnatives-ursa-...
Re: Ursa: A leaderless, object storage–based alternative to Kafka
#20Was the key unlock here the ability to append data to an object? ( https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... )
There are a few things unlocked by Ursa: 1. It is leaderless by design. So there is no single lead broker you need to route the traffic. So you can eliminate majority of the inter-zone traffic. 2. It is lakehouse-native by design. It is not only just use object storage as the storage layer, but also use open table formats for storing data. So streaming data can be made available in open table formats (Iceberg or Delt…
They were asking about changes that enabled Ursa itself.