Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 329 posts

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#11

Post by huntaub »

krawczstef wrote: post #9

Does this compete with Minio?

I don't think so, I see them as complementary. MinIO is great when you have downstream applications which speak the S3 API that need acceleration of that data. Regatta is designed for applications which speak file semantics (think, application logging, storing corpuses of training data, or state) that doesn't run on the S3 API. Regatta actually supports MinIO as an S3-compatible backend for your file system!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#12

Post by foft »

Interesting. Reminds me of FlexFS (https://flexfs.io/). I spoke to a very knowledgeable person there when investigating what to use but we ended up using EFS instead.

An annoying feature of EFS is how it scales with amount of storage, so when its empty its very slow. We also started hitting its limits so could not scale our compute workers. Both can be solved by paying for the elastic iops but that is VERY expensive.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#15

Post by huntaub »

remram wrote: post #13

Is this like JuiceFS? https://juicefs.com/

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using it's native format -- so you can do these things!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#16

Post by huntaub »

mbrumlow wrote: post #14

Is every file a s3 object? What if you change the middle of a large file?

That's correct -- every file is an S3 object. If you change the middle of a large file, Regatta will store the change on our durable caching layer efficiently (and most writes complete in under 1ms). Regatta will then asynchronously update the large object in S3, which may take longer. We automatically batch multiple changes together to minimize the number of operations to your S3 bucket!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#17

Post by mr90210 »

That’s so nice see, because in the few days I had been tinkering with the concept of file system + blob storage but I had hard time com up with use-cases other than an unlimited Dropbox where you own the storage and truly pay as you go.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#18

Post by huntaub »

foft wrote: post #12

Interesting. Reminds me of FlexFS ( https://flexfs.io/ ). I spoke to a very knowledgeable person there when investigating what to use but we ended up using EFS instead. An annoying feature of EFS is how it scales with amount of storage, so when its empty its very slow. We also started hitting its limits so could not scale our compute workers. Both can be solved by paying for the elastic iops but that is VERY expensiv…

Yes, I think it's similar product, but we're looking to provide high performance on all dimensions (latency, throughput, and IOPS). I totally agree with you that Elastic Throughput solves this problem, but it can be expensive for many workloads!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#19

Post by crest »

jeffbee wrote: post #7

People have been throwing out "POSIX" distributed file systems for a long time but this claim usually raises more questions than it answers. Especially since clients access it via NFSv3, which has extremely weak semantics and leaves most POSIX filesystem features unimplemented.

You can implement a single client NFSv3 server that provides stronger than expected (of NFSv3) guarantees and if you implement the "optional" companion protocols it should come closer to local filesystem semantics than most network filesystems. What would be neat about such a solution is that you can run the server either locally or remotely (same site, high bandwidth, low latency) and at the same time clients would have to a custom FUSE server or even worse load an (from the customer's point of view) experimental vendor kernel module. Upgrading from NFSv3 to NFSv4 would get you a bit closer to POSIX semantics, but of course it would still be NFS just not over a congested, jittery link to a shared server. Especially NFSv4 delegations could be a nice way to let the clients kernel buffer a lot of bursty async I/O locally. Just keep in mind how little POSIX really guarantees instead of assuming it will behave like ext4/XFS or even better ZFS on a laptop NVMe with two levels of power loss protection (big caps in the drive and the laptop battery).
Post reply on HN