Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 329 posts

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#201

Post by benatkin »

huntaub wrote: post #191
benatkin wrote: post #190

Earlier quoted context omitted.

The headline seems misleading, then. rclone can work with AWS' different offerings, some of which at least partially address this: https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-z...

I'm not totally sure what you mean. I don't think that S3 Express One Zone offers any additional atomic semantics in the file system world.

For the misleading part, I probably should have said confusing because I don't think you intended that, I mean that instead of introducing your caching layer you make it about S3, where the Object Storage provider seems totally interchangeable. Though it seems to work for a lot of your audience, from what I can tell from other comments here.

As for Express One Zone providing consistency, it would make more groups of operations consistent, provided that the clients could access the endpoints with low latency. It wouldn't be a guarantee but it would be practical for some applications. It depends on what the problem is - for instance, do you want someone to never see noticeably stale data? I can definitely see that happening with Express One Zone if it's as described.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#202

Post by huntaub »

benatkin wrote: post #201
huntaub wrote: post #191

Earlier quoted context omitted.

I'm not totally sure what you mean. I don't think that S3 Express One Zone offers any additional atomic semantics in the file system world.

For the misleading part, I probably should have said confusing because I don't think you intended that, I mean that instead of introducing your caching layer you make it about S3, where the Object Storage provider seems totally interchangeable. Though it seems to work for a lot of your audience, from what I can tell from other comments here. As for Express One Zone providing consistency, it would make more groups of…

Yes, I think this is something that I’m actually struggling with. What’s the most exciting part for users? Is it the fact that we’re building a super fast file system or is it that we have this synchronization to S3? Ultimately, there just isn’t space for it all — but I appreciate the feedback.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#203

Post by sieabahlpark »

huntaub wrote: post #40
memset wrote: post #35

This is honestly the coolest thing I've seen coming out of YC in years. I have a bunch of questions which are basically related to "how does it work" and please pardon me if my questions are silly or naive! 1. If I had a local disk which was 10 GB, what happens when I try to contend with data in the 50 GB range (as in, more that could be cached locally?) Would I immediately see degradation, or thrashing, at the 10 GB…

Wow, thanks for the nice note! No questions are silly, and I'll also note that we now have a docs site ( https://docs.regattastorage.com ) and feel free to email me (hleath [at] regattastorage.com) if I don't fully address your questions. > If I had a local disk which was 10 GB, what happens when I try to contend with data in the 50 GB range (as in, more that could be cached locally?) Would I immediately see degradat…

[dead]

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#204

Post by huntaub »

hanslovsky wrote: post #198

that looks interesting. we spent a lot of money on FSxL and might save a lot with Weka. Unfortunately, our data access pattern is very random and will likely not benefit from caching unless we cache the entire dataset (100TB)

One thing that we’re considering for these kinds of use cases is providing the ability for users to either (a) request that their data always stays in the hot storage or (b) provide the ability for users to ask us to pre-load their data sets when they begin analysis (and I expect that our large fleet of instances can do this preloading much quicker than any individual instance download). Would either of those options make the product more helpful for you?

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#205

Post by huntaub »

harshaw wrote: post #196
senderista wrote: post #192

If this product is successful, what prevents AWS from cloning it at a lower price (perhaps by leveraging access to their infrastructure) and putting you out of business?

There is room in storage for many kind of products with different sets of tradeoffs. And don't underestimate the ability of a startup to move much faster than AWS.

For the most part, I agree with this! The storage market is big enough for everyone because of all of the different needs that customers have. That said, if we are able to build something that provides EBS-like performance with pay-as-you-go pricing, there are likely large economic reasons why AWS would not chase making that the default.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#206

Post by huntaub »

gjmveloso wrote: post #195

Feels like FSx for Lustre without the complexity. Definitely what EFS could be. Congrats on the launch!

Thank you! That’s exactly my hope! How can we make these technologies as easy as possible for teams to use so that they can build the best or whatever applications without becoming storage experts.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#207

Post by sieabahlpark »

wongarsu wrote: post #185
nine_k wrote: post #155

Earlier quoted context omitted.

Hiring someone who knows how to manage bare metal (with failover and stuff) may take time %)

You pay a datacenter to put it in a rack and add connect power and uplinks, then treat it like a big ec2 instance (minus the built-in firewall). Now you just need someone who knows how to secure an ec2 instance and run your preferred software there (with failover and stuff). If you run a single-digit number of servers and replace them every 5 years you will probably never get a hardware failure. If you're unlucky and…

[dead]

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#208

Post by huntaub »

bx376 wrote: post #199

Sounds similar to https://juicefs.com/

It is! Let me share some details in differences that I’ve posted elsewhere:

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using its native format -- so you can do these things!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#209

Post by benatkin »

huntaub wrote: post #202
benatkin wrote: post #201

Earlier quoted context omitted.

For the misleading part, I probably should have said confusing because I don't think you intended that, I mean that instead of introducing your caching layer you make it about S3, where the Object Storage provider seems totally interchangeable. Though it seems to work for a lot of your audience, from what I can tell from other comments here. As for Express One Zone providing consistency, it would make more groups of…

Yes, I think this is something that I’m actually struggling with. What’s the most exciting part for users? Is it the fact that we’re building a super fast file system or is it that we have this synchronization to S3? Ultimately, there just isn’t space for it all — but I appreciate the feedback.

I think they both go together. It might take about 10 minutes to give a good high level explanation of it, including how the S3 syncing works - that the S3 lags slightly behind the caching layer for reads, and that you can still write to S3. 2-way sync. I imagine that S3 would be treated sort of like another client if updates came from S3 and the clients at the same time. It would probably be not so great to write to S3 if you aren't writing to somewhere that's being actively edited, but if you want to write to a dormant area of S3 directly, that's fine.
Post reply on HN