Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ursa: A leaderless, object storage–based alternative to Kafka

streamnative.io

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#21

Post by Imustaskforhelp »

sijieg wrote: post #12
codeaether wrote: post #3

License? It doesn't seem to be open sourced.

I am one of the co-founders of StreamNative. Currently Ursa is only available in our cloud service. But we do plan to open-source the core soon. Stay tuned.

Can't wait for you guys to open source this stuff. If I may ask, what's the license you guys are thinking of? Since I am interested in hoping to someday live as a developer while working on open source too but its a tough line b/w getting no sponsors with MIT license and being called non foss and being charged in HN for some crimes because you used some license like SSPL or some custom license.

The sad reality is that most people in open source want stuff for free and won't pay back and that sucks. So what are your thoughts on this? I am genuinely curious.

The second part as someone noted, in a comment of the parent comment that you are responding, that code is not the most important part here, how much do you agree with that statement? Since to me, If I can self host it using open source without using your cloud service but rather using amazon directly, I do think that might be cheaper than using the cloud service directly.

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#22

Post by Imustaskforhelp »

oulipo wrote: post #9

If it's not open-source or at least self-hosteable I don't think it will be that useful

Not a part of Ursa but I think that they are hoping to do so in the future. Usefulness can come later, I am more than happy to wait in the meanwhile

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#23

Post by Imustaskforhelp »

Lets hope that you guys open source this in a great manner and actually still live really nicely.

If I may ask a philosophical question, when would you consider your product to "succeed", would it be when someone uses it for something important or some money related benchmark or what exactly

Wishing Ursa team peace and success. maybe don't ever enshittify your product as so many do. Will look at you from the sidebars since I don't have a purpose to even kafka but I would recommend having some discord or some way to actually form a community I suppose. I recommend matrix but there are folks who are discord too.

Anyways, have fun building new things!

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#24

Post by x0x0 »

davidkj wrote: post #14
x0x0 wrote: post #5

Earlier quoted context omitted.

As near as I can tell, the claims of huge cost savings derive from the difficulty dynamically scaling Kafka and improved multitenancy. So if different pieces of your company each have overprovisioned kafka clusters, they could all move to Ursa and save all the overprovisioning. I have not tried it, and full disclosure, I really like Kafka: it's one of the pieces of software that has been rock solid for me. I built a…

The bulk of the cost savings comes from the use of object storage rather than attached disks. This eliminates the inter-AZ networking costs associated with Kafka replication mechanism. I break all of the costs down in the following e-book. https://streamnative.io/ebooks/reducing-kafka-costs-with-lea...

So basically Kafka, to provide availability guarantees, requires multi-AZ and the inter-AZ replication gets expensive. And Ursa avoids that by using object storage and probably then just talking inter-AZ?

And while I like Kafka, nobody would claim it likes being scaled up and down dynamically, so probably built-in tolerance for that as well? We ran Kafka on-prem so that wasn't an issue for us, and given the nature of the service, didn't have a lot of usage variance.

This: https://www.youtube.com/watch?v=bb-_4r1N6eg was an interesting watch, btw.

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#25

Post by akshayshah »

rohan_ wrote: post #8

Was the key unlock here the ability to append data to an object? ( https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... )

No, it was S3 becoming strongly consistent in 2020: https://www.infoq.com/news/2020/12/aws-s3-strong-consistency...

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#26

Post by _benedict »

Do you anywhere elaborate what you mean by leaderless, and how this affects the semantics and guarantees you offer?

So far as I understand both Kafka and Pulsar use (leader-based) consensus protocols to deliver some of their features and guarantees, so to match these you must either have developed a leaderless consensus protocol, or modify the guarantees you offer, or else have a leader-based consensus protocol you utilise still?

From one of your other answers, you mention you rely on Apache Bookkeeper, which appears to be leader-based?

I ask because I am aware of only one industry leaderless consensus protocol under development (and I am working on it), and it is always fun to hear about related work.

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#27

Post by supermatt »

rohan_ wrote: post #8

Was the key unlock here the ability to append data to an object? ( https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... )

That’s probably not as useful as you think. Unless things have changed more recently, you need to set the offset from which to append, which makes it near useless for most use cases where appending would actually be useful.

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#29

Post by jauntywundrkind »

wmal wrote: post #28

How does it compare to AutoMQ? ( https://github.com/AutoMQ/automq )

AutoMQ look so so promising. Very happy to see the shift to Apache 2.0 license a couple month ago!! I do think it sounds like the most obvious comparison to Ursa: object-storage based, focus on removing inter-zone traffic. They also have a neat new Table Topics, that's super helpful. https://www.automq.com/docs/automq/eliminate-inter-zone-traf...

There's an OK high level cruise, WarpStream is dead, long live AutoMQ riffing off WarpStream doing similar against Kafka. While I loosely got the idea, I had to dig a lot deeper in docs for things to start to really click. https://github.com/AutoMQ/automq/wiki/WarpStream-is-dead,-lo...

There may be reasons it's a bad fit, but I'm expecting object-storage database SlateDB someday makes a very fine streaming system too!! https://github.com/slatedb/slatedb

Re: Ursa: A leaderless, object storage–based alternative to Kafka

#30

Post by deniscoady »

codelipenghui wrote: post #7

Just share a blog post published before, which compares the costs of running a 5 GB/s Kafka workload using Ursa, Warpstream, MSK, and Redpanda: https://streamnative.io/blog/how-we-run-a-5-gb-s-kafka-workl... And the test result was verified by Databricks: https://www.linkedin.com/posts/kramasamy_incredible-streamna... The analysis in the blog is based on two key assumptions: - Multi-zone deployment on AWS - Tiered st…

I'm an employee at Redpanda.

> Redpanda recently introduced leader pinning, but this only benefits setups where producers are confined to a single AZ—not applicable to our multi-AZ benchmark.

Redpanda has leadership pinning (producers) and follower fetching (consumers). I suspect a significant amount of cost is improper shaping of traffic.

> Interzone traffic - replication: 10GB/s * $0.02/GB(in+out) * 3600 = $720

With follower fetching you shouldn't have cross-AZ charges on read, only on replication. In 15 seconds of looking at this piece I cut out $360/hour...no offense but this reeks of bad faith benchmarketing...

Post reply on HN