Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 329 posts

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#141

Post by garganzol »

I used the same approach based on Rclone for a long time. I wondered what makes Regatta Storage different than Rclone. Here is the answer: "When performing mutating operations on the file system (including writes, renames, and directory changes), Regatta first stages this data on its high-speed caching layer to provide strong consistency to other file clients." [0].

Rclone, on the contrary, has no layer that would guarantee consistency among parallel clients.

[0] https://docs.regattastorage.com/details/architecture#overvie...

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#142

Post by huntaub »

nwgo wrote: post #136

Is there any open source alternative to something like this?

Hey, thanks for asking. It very much depends on which aspect of Regatta you're interested in using. I know of a couple of different architectures -- some folks wrote in "rclone" in the thread, I know of people using SeaweedFS if you want to host storage infrastructure yourself, etc.

I'd love to know a bit more about why you're looking for an open source alternative. Is it because of costs (i.e. you'd like an open source alternative that doesn't require you to pay) or if it's because of the operating environment (i.e. you want an open source alternative so that you can deploy it to your own infrastructure)? There are some things that we are exploring around deploying onto your own infrastructure over the next 12 months, but I'd love to learn more. Feel free to respond here or email me at hleath [at] regattastorage.com.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#143

Post by huntaub »

geophile wrote: post #137

How does this differ from what Nasuni offers?

Hey there, I have mutual friends with some of the Nasuni folks, and I have a lot of respect for what they do. In particular, Nasuni stores data in a proprietary block format in your S3 bucket, so you can't connect it to existing data sets or use that data directly from S3 out the other side. Whereas with Regatta, we store data in its native format in S3 so you can do these things.

What's cool about the storage market is that there are so many impressive companies because there are so many varied needs from customer applications! We're hoping to become a simple "default" for teams who are writing applications in the cloud.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#144

Post by convivialdingo »

Wow, looks like a great product! That's a great idea to use NFS as the protocol. I honestly hadn't thought of that.

Perfect.

For IBM, I wrote a crypto filesystem that works similarly in concept, except it was a kernel filesystem. We crypto split the blocks up into 4 parts, stored into cache. A background daemon listened to events and sync'ed blocks to S3 orchestrated with a shared journal.

It's pure magic when you mount a filesystem on clean machine and all your data is "just there."

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#145

Post by random3 »

In (March?) 2007 (correction 2008) myself and two other engineers in front of Bruce Chizen - Adobe's CEO in a small conference room in Bucharest demoed a photo taken with an iPhone automagically showing as a file on a Mac. I implemented the local FUSE talking to Ozzy - Adobe's distributed object store back then, using an equivalent of a Linux inode structure. It worked like a charm and if I remember correctly it took us a few days to build it. It was a success just as much as Adobe's later choices around http://Photoshop.com were a huge failure. A few months later Dropbox launched.

That kickstarted about a decade in (actual) research and development led by my team which positioned the Bucharest center as one of the most prolific centers in distributed systems within Adobe and of Adobe within Romania.

But I didn't come up with the concept, it was Richard Jones that inspired us with the Gmail drive that used FUSE with gmail attachments back in 2004 when I got my first while still in college https://en.wikipedia.org/wiki/GMail_Drive. I guess I'm old, but I find it funny to see Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#146

Post by garganzol »

Super interesting project. But I cannot understand why you support only EC2 instances as clients. For what it is worth, it looks strange and limiting. By default I expect to be able to use Regatta Storage from everywhere: from my local machine, from my Docker containers running elsewhere, etc.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#148

Post by huntaub »

garganzol wrote: post #141

I used the same approach based on Rclone for a long time. I wondered what makes Regatta Storage different than Rclone. Here is the answer: "When performing mutating operations on the file system (including writes, renames, and directory changes), Regatta first stages this data on its high-speed caching layer to provide strong consistency to other file clients." [0]. Rclone, on the contrary, has no layer that would gu…

This is exactly right, and something that we think is particularly important for applications that care about data consistency. Often times, we see that customers want to be able to quickly hand off tasks from one instance to another which can be incredibly complex if you don't have guarantees that your new operations will be seen by the second instance!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#149

Post by huntaub »

convivialdingo wrote: post #144

Wow, looks like a great product! That's a great idea to use NFS as the protocol. I honestly hadn't thought of that. Perfect. For IBM, I wrote a crypto filesystem that works similarly in concept, except it was a kernel filesystem. We crypto split the blocks up into 4 parts, stored into cache. A background daemon listened to events and sync'ed blocks to S3 orchestrated with a shared journal. It's pure magic when you mo…

> It's pure magic when you mount a filesystem on clean machine and all your data is "just there."

I totally agree! I am hoping that Regatta can power a future where teams don't need more than ~8 GiB of local storage for their operating system, and can store the rest on something like Regatta to get rid of the waste of overprovisioned block volumes.

Post reply on HN