Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 329 posts

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#51

Post by huntaub »

count wrote: post #45

I don't see any other question about it, so maybe I just missed the obvious answer, but how do you handle POSIX ACLs? If the data is stored as an object in S3, but exposed via filesystem, where are you keeping (if at all?) the filesystem ACLs and metadata? Also, NFSv3 and not 4?

Great call out. Some kinds of data, like ACLs and specific kinds of metadata, don't live in S3. Full disclosure, we don't support ACLs today (but plan to soon). We keep file system metadata in the durable cache. For some files (where users haven't changed permissions, etc), we are able to release that cached metadata when the file is no longer in use. For other files (where permissions have been changed by the user), that metadata must live in the cache long-term.

We selected NFSv3 due to it's broad compatibility with different compute environments. For example, Windows has an NFSv3 client in it, but doesn't have an NFSv4 client. There are lots of enterprise workloads which needs simultaneous access to file data from both Windows and Linux, and supporting NFSv3 was the easiest path to support those workloads.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#52

Post by huntaub »

mikecwang wrote: post #50

Does it mean I can use Lambda + SQLite + Regatta to build a real pay-as-you-go ACID SQL storage? Edit: an production-ready (high durability) ACID SQL storage

Yes! This is my expectation. Lots of the big companies have already done this with in-house architecture. With Regatta, we want to democratize building stateless applications that can take advantage of the low-cost storage of S3.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#54

Post by mikecwang »

huntaub wrote: post #52
mikecwang wrote: post #50

Does it mean I can use Lambda + SQLite + Regatta to build a real pay-as-you-go ACID SQL storage? Edit: an production-ready (high durability) ACID SQL storage

Yes! This is my expectation. Lots of the big companies have already done this with in-house architecture. With Regatta, we want to democratize building stateless applications that can take advantage of the low-cost storage of S3.

That's some real tech in YC these days!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#55

Post by huntaub »

bithive123 wrote: post #48

How does this compare to Amazon's own offering in this space, the "AWS Storage Gateway"? It can also back various storage protocols with S3, using SSDs for cache, etc. ( https://aws.amazon.com/storagegateway/features/ )

Great question! We fill the same role as AWS Storage Gateway (and I used to work closely with that team when I was at AWS, lots of respect for what they do). AWS Storage Gateway is built primarily as an appliance to be installed on instances in your own data center to ease migration to the cloud. Many customers do deploy Storage Gateway on EC2 because they want these features in the cloud itself. However, the "appliance" design of Storage Gateway makes it unsuitable for this purpose. For example, Storage Gateway is not designed to run in a cluster for high-availability and doesn't have access to durable, long-term storage to stage and cache writes.

On the other hand, Regatta is designed as a cloud-native gateway product. Regatta's elastic, durable caching layer allows us to efficiently cache large data sets without thrashing, and always efficiently perform writes. Because Regatta is designed to be highly-available, customers don't have to worry about downtime for patching or deployments.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#56

Post by renewiltord »

Fascinating. If this had been around a year ago, we could have used it in our datacenter build-out. For data source reasons, we record data in the cloud. In the past, we'd stick most of the data in S3 and only egress what we needed to run analysis on. The way we'd do that is that we have a machine with 16 * 30 TiB SSDs that acts as our on-prem cache of our S3 data. It did this using a slightly modified goofys with a more modified catfs in front of it, with both the cache and the catfs view exported over NFSv4. We had application-level switching between the cache and the export since our data was really read-only.

When the cache got full, catfs would evict things from it pretty simply. It's overall got a good design but has a few bugs you have to fix, and when you have 100 machines connecting to it, it requires some tuning to make sure that it doesn't all stall. But it worked for the most part.

Anyway, I think this is cool tech. I'm currently doing some bioinformatics stuff that this might help with (each genome sequence is some 100 GiB compressed). I'll give it a shot some time in the next couple of months.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#57

Post by paulgb »

huntaub wrote: post #6
koolba wrote: post #4

Neat stuff. I think everybody with an interest in NFS has toyed with this idea at some point. > Under the hood, customers mount a Regatta file system by connecting to our fleet of caching instances over NFSv3 (soon, our custom protocol). Our instances then connect to the customer’s S3 bucket on the backend, and provide sub-millisecond cached-read and write performance. This durable cache allows us to provide a strong…

Thanks for the question! > How do you handle the cache server crashing before syncing to S3? Do the cache servers have local disk as well? Our caching layer is highly durable, which is (in my opinion) the key for doing this kind of staging. This means that once a write is complete to Regatta, we guarantee that it will eventually complete on S3. For this reason, server crashes and intermittent S3 availability issues a…

Congrats on the launch, this is really cool! Is the durable cache an attached disk, or are you using a separate AWS product for that?

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#58

Post by huntaub »

cluckindan wrote: post #46

How does this compare to S3 compatible CSI drivers like DirectPV?

I could totally be misreading DirectPV, but it appears to be a way to use K8s Persistent Volumes to manage things like NVME drives which are attached to each node, and doesn't provide any tie in to S3 (outside of the fact that it's built to power MinIO).

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#59

Post by huntaub »

Shoop wrote: post #39
huntaub wrote: post #16

Earlier quoted context omitted.

That's correct -- every file is an S3 object. If you change the middle of a large file, Regatta will store the change on our durable caching layer efficiently (and most writes complete in under 1ms). Regatta will then asynchronously update the large object in S3, which may take longer. We automatically batch multiple changes together to minimize the number of operations to your S3 bucket!

What are the consistency semantics?

All connected file system clients see strong, read-after-write consistency. Most file operations are synchronized to S3 within a few minutes of completion.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#60

Post by huntaub »

paulgb wrote: post #57
huntaub wrote: post #6

Earlier quoted context omitted.

Thanks for the question! > How do you handle the cache server crashing before syncing to S3? Do the cache servers have local disk as well? Our caching layer is highly durable, which is (in my opinion) the key for doing this kind of staging. This means that once a write is complete to Regatta, we guarantee that it will eventually complete on S3. For this reason, server crashes and intermittent S3 availability issues a…

Congrats on the launch, this is really cool! Is the durable cache an attached disk, or are you using a separate AWS product for that?

Without getting too much into the details of the system, our durable cache is designed for 5 9s of durability (and we're working on a version that will provide 11 9s of durability soon). You can't achieve those durability numbers on a single attached NVMe device without some kind of replication.
Post reply on HN