I realize it isn't your target use case, but I'm tempted to move all of my personal stuff stored in Google Drive over to this.
Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
251–260 of 329 posts
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#252Earlier quoted context omitted.
Thank you for the detailed answers! Honestly, this project inspires me to work on infrastructure problems. So you are saying that regatta's own SaaS infrastructure provides the disk caching layer. So you all make sure the pipe between my AWS instance and your servers are very fast and "infinitely scalable", and then the sync to S3 happens after the fact.
That's exactly right!
So Regatta has an in memory cache? Does the posix disk write only suceed when the data is in more than one availability zone?
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#253Founder of cunoFS here, brilliant to see lots of activity in this space, and congrats on the launch! As you'll know, there's a whole galaxy of design decisions when building file storage, and as a storage geek it's fun to see what different choices people make!
I see you've made some similar decisions to what we did for similar reasons I think - making sure files are stored 1:1 exactly as an object without some proprietary backend scrambling, offering strong consistency and POSIX semantics on the file storage, with eventual consistency between S3 and POSIX interfaces, and targeting high performance. Looks like we differ on the managed service vs traditional download and install model, and the client-first vs server-first approach (though some of our users also run cunoFS on an NFS/SMB gateway server), and caching is a paid feature for us versus an included feature for yours.
Look forward to meeting and seeing you at storage conferences!
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#254Earlier quoted context omitted.
Yes! This is my expectation. Lots of the big companies have already done this with in-house architecture. With Regatta, we want to democratize building stateless applications that can take advantage of the low-cost storage of S3.
Wouldn't that limit the concurrency of the lambdas to 1? Since they would hold a lock on the db file
Well these are the details that many of us is interested.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#255Earlier quoted context omitted.
Hiring someone who knows how to manage bare metal (with failover and stuff) may take time %)
You pay a datacenter to put it in a rack and add connect power and uplinks, then treat it like a big ec2 instance (minus the built-in firewall). Now you just need someone who knows how to secure an ec2 instance and run your preferred software there (with failover and stuff). If you run a single-digit number of servers and replace them every 5 years you will probably never get a hardware failure. If you're unlucky and…
We run on our own stuff at our shop.
Some things that are hidden in the cloud providers cost are redundant networking, redundant internet connection, redundant disks.
Likely still cheaper than the cloud obviously but you will need to stomach down time for that stuff if something breaks.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#256Founder of cunoFS here, brilliant to see lots of activity in this space, and congrats on the launch! As you'll know, there's a whole galaxy of design decisions when building file storage, and as a storage geek it's fun to see what different choices people make! I see you've made some similar decisions to what we did for similar reasons I think - making sure files are stored 1:1 exactly as an object without some propr…
Great to hear from you, I think cunoFS is doing a lot of things right! It’s certainly a fun problem space!
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#257I used the same approach based on Rclone for a long time. I wondered what makes Regatta Storage different than Rclone. Here is the answer: "When performing mutating operations on the file system (including writes, renames, and directory changes), Regatta first stages this data on its high-speed caching layer to provide strong consistency to other file clients." [0]. Rclone, on the contrary, has no layer that would gu…
I suppose rclone doesn't provide byte range file locking? Running sqlite over rclone would be a disaster.
That would be my expectation, you need something in the middle to actually broker the file locks.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#258I’m very interested in this as a backing disk for SQLite/DuckDB/parquet, but I really want my cached reads to come straight from instance-local NVMe storage, and to have a way to “pin” and “unpin” some subdirectories from local cache. Why local storage? We’re going to have multiple processes reading & writing to the files and need locking & shared memory semantics you can’t get w/ NFS. I could implement pin/unpin mys…
Hey -- I think this is something that's in-scope for our custom protocol that we're working on. I'd love to chat more about your needs to make sure that we build something that will work great for you. Would you mind shooting an email to hleath [at] regattastorage.com and we can chat more?
We're also interested in SQLite shared by multiple processes on something like Regatta but my concerns are the issues described in the SQLite documentation about NFS [1]. Notably "SQLite relies on exclusive locks for write operations, and those have been known to operate incorrectly for some network filesystems."
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#259Earlier quoted context omitted.
That's exactly right!
So Regatta has an in memory cache? Does the posix disk write only suceed when the data is in more than one availability zone?
Hey there! Today, we are replicating cache data within a single availability zone, but we’re working on a multi-availability zone product. If you have a need for multi-AZ, please shoot me an email at hleath [at] regattastorage.com, I’d love to learn more
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#260How do you handle wrote concurrency? If you different processes write on the same file at the same time, what do I read after?
All connected file system clients see read-after-write consistency, so you see the up to date file data!