Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

news.ycombinator.com

Post reply on HN
Print view View original

321–329 of 329 posts

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#321

Post by loufangxin »

huntaub wrote: post #15
remram wrote: post #13

Is this like JuiceFS? https://juicefs.com/

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using it's native format -- so you can do these things!

After reading the document, I think "proprietary block format" is a design. Obviously this design can provide parallel data flush back to S3 quickly, and we can cache the reading block and prefetch the next block for more cache friendly.

Seems that JuiceFS have S3 gateway, so you can still access the files in S3 protocol.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#322

Post by craigkilgo »

huntaub wrote: post #308
craigkilgo wrote: post #295

How did you choose the name Regatta?

I spent several years working in the Northeast, and I developed an appreciation (but not a skill) for sailing. In some sense, I think of Regatta as high-speed sailing on top of customer data lakes.

Sailing, I was hoping it was a rowing metaphor.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#323

Post by amitizle »

huntaub wrote: post #228
amitizle wrote: post #226

Why are these solutions always using NFS? I'm asking out of curiosity, not judgement. I've looked for a solution to write many small files fast (safely). Think about cloning thr Linux kernel git repo. Whatever I tested, the NFS protocol was always a bottleneck.

We choose NFS purely because it's the fastest way to get broad compatibility with most operating systems (NFSv3, for example is supported on both Linux and Windows). However, I have great news for you! We're simultaneously working on a custom protocol (over FUSE today) that is going to solve the small file problem for things like cloning the Linux kernel git repo. You can actually see in our demo video ( https://yout…

Great! Thanks!

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#324

Post by remram »

loufangxin wrote: post #321
huntaub wrote: post #15

Earlier quoted context omitted.

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using it's native format -- so you can do these things!

After reading the document, I think "proprietary block format" is a design. Obviously this design can provide parallel data flush back to S3 quickly, and we can cache the reading block and prefetch the next block for more cache friendly. Seems that JuiceFS have S3 gateway, so you can still access the files in S3 protocol.

What do you mean by "is a design"?

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#325

Post by cuno »

Andys wrote: post #284
cuno wrote: post #253

Founder of cunoFS here, brilliant to see lots of activity in this space, and congrats on the launch! As you'll know, there's a whole galaxy of design decisions when building file storage, and as a storage geek it's fun to see what different choices people make! I see you've made some similar decisions to what we did for similar reasons I think - making sure files are stored 1:1 exactly as an object without some propr…

Is that Gweo? Didn't know you were in the storage space, good to see you!

Good to see you too! Lets catchup sometime - tried to connect with you separately but no luck.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#327

Post by foodbaby »

renewiltord wrote: post #56

Fascinating. If this had been around a year ago, we could have used it in our datacenter build-out. For data source reasons, we record data in the cloud. In the past, we'd stick most of the data in S3 and only egress what we needed to run analysis on. The way we'd do that is that we have a machine with 16 * 30 TiB SSDs that acts as our on-prem cache of our S3 data. It did this using a slightly modified goofys with a…

Super interesting. What did you have to modify and tune in goofyfs and catfs? Did you consider using Lustre at all?

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#328

Post by foodbaby »

huntaub wrote: post #206
gjmveloso wrote: post #195

Feels like FSx for Lustre without the complexity. Definitely what EFS could be. Congrats on the launch!

Thank you! That’s exactly my hope! How can we make these technologies as easy as possible for teams to use so that they can build the best or whatever applications without becoming storage experts.

How does it compare perf-wise to FSX Lustre backed by S3? Other than pay-as-you-go are there any functional differences?

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#329

Post by huntaub »

foodbaby wrote: post #328
huntaub wrote: post #206

Earlier quoted context omitted.

Thank you! That’s exactly my hope! How can we make these technologies as easy as possible for teams to use so that they can build the best or whatever applications without becoming storage experts.

How does it compare perf-wise to FSX Lustre backed by S3? Other than pay-as-you-go are there any functional differences?

Ultimately, we should be able to match the performance of FSx for Lustre. With a scale-out protocol (mirroring the Lustre protocol), it's relatively simple to achieve scaleable throughput and IOPS out to the level of 100s of GiB/s and millions of IOPS (we aren't doing this today, but I anticipate being here within a month or two).

Functionally, we're not running DRTs like Lustre does in order to synchronize from S3. We will support the ability to pre-load the cache (like a DRT), for customers who can't handle the lazy-loading latency, but unlike Lustre, we can't run out of capacity in the hot tier, and we will automatically delete data from the hot tier when it's not in use. I know that DRT management can be complex for customers.

Post reply on HN