Ok that's cool but like... you could've just given me a bashscript to do the same thing instead of the pitchdeck-followup baggage of the n-th try at recreating the dropbox lottery shot from a decade and a half ago...
Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
311–320 of 329 posts
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#312Interesting. Reminds me of FlexFS ( https://flexfs.io/ ). I spoke to a very knowledgeable person there when investigating what to use but we ended up using EFS instead. An annoying feature of EFS is how it scales with amount of storage, so when its empty its very slow. We also started hitting its limits so could not scale our compute workers. Both can be solved by paying for the elastic iops but that is VERY expensiv…
Yes, I think it's similar product, but we're looking to provide high performance on all dimensions (latency, throughput, and IOPS). I totally agree with you that Elastic Throughput solves this problem, but it can be expensive for many workloads!
First, Regatta sounds extremely helpful, and I’ve enjoyed reading your responses.
Responding here to say that I’d love to hear more about your comparison to FlexFS. In fact, I’d love to see a few of them: FlexFS, MountPoint, etc
Lastly, I couldn’t get the privacy policy to load on your site (I’m on mobile if that helps)
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#313Earlier quoted context omitted.
Regatta is a write-through cache for s3 bucket under its supervision? I guess then external changes to that bucket is a no-no? Any plans to expand to other stores, like R2 (I ask since unlike S3, R2 egress is free)?
Hey there, that's sort of the correct way to think about it -- notably that our caching layer is high-durability, so we can keep recent writes in the cache safely. External changes to the bucket are okay! Lots of customers need to (for example) ingest data into S3, then process it on a file system, and that totally works. The only thing that isn't supported is editing the same file from both S3 and the file system si…
I actually asked about R2 to see if Regatta's pricing is any different as there's no egress fee. I should have been clearer.
btw, thanks a bunch for answering my Q & everyone else's too (except for parts where you couldn't talk about the implementation, understandably so). Appreciate it. Wishing the best.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#314[dead]
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#315Ok that's cool but like... you could've just given me a bashscript to do the same thing instead of the pitchdeck-followup baggage of the n-th try at recreating the dropbox lottery shot from a decade and a half ago...
That’s true, we could just release the software open source, but that doesn’t help our customers who don’t want to run and manage their own infrastructure. Our customers tell us that the value of the product comes from it being fully managed — they simply need to click a button, and all of this works out of the box.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#316Earlier quoted context omitted.
Is there something specific that you worry about when running a database on a networked file system? I would imagine that any database which is correctly fsync'ing the data to the write-ahead-log should work just fine.
First of all databases don't support running on NFS. It is an unsupported configuration. The deeper reason for that is, that the consistency guarantees from NFS (close-to-open consistency) are a lot weaker than what you get from POSIX.
I don’t know if I agree, for example, Postgres has this [1] to say about using NFS as the backing store. I think that part of the challenge is that there are so many implementation details that differ between NFS servers and many configuration options that teams can fiddle with (Postgres specifically calls out “async” as dangerous). Close to open semantics are actually stronger than what something like XFS offers (because XFS isn’t required to flush data to disk on file close), and databases should be fsyncing their write ahead logs from the application layer. Like said though, this doesn’t mean that there aren’t certain configurations of NFS which won’t work (async for example means that NFS servers won’t actually write to non-volatile storage on fsync, which is of course dangerous for any application).
[1] https://www.postgresql.org/docs/current/creating-cluster.htm...
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#317Earlier quoted context omitted.
Yes, I think it's similar product, but we're looking to provide high performance on all dimensions (latency, throughput, and IOPS). I totally agree with you that Elastic Throughput solves this problem, but it can be expensive for many workloads!
First, Regatta sounds extremely helpful, and I’ve enjoyed reading your responses. Responding here to say that I’d love to hear more about your comparison to FlexFS. In fact, I’d love to see a few of them: FlexFS, MountPoint, etc Lastly, I couldn’t get the privacy policy to load on your site (I’m on mobile if that helps)
Thank you for the note. I’d recommend checking out this section of our docs [1], where we are trying to compile some of this comparison. I haven’t called out FlexFS specifically, but I’ll work on adding that soon. We’ll also get the Privacy Policy fixed today, thanks for pointing that out.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#318Earlier quoted context omitted.
Are you going to risk your entire business over "probably never get a hardware failure" that, if it hits, might result in days of downtime to resolve? I wouldn't.
Just pay 2x for the hardware and have a hot standby, 1990s-style. Practice switching between the boxes every month or so; should be imperceptible for the customers and a nearly non-event for the ops.
How many hours of labor does that take every month you failover? What about hot hard drive spares? Do you want networking redundancy? How about data backups? Second set of hot servers in another physical data center?
All of that costs money and time. You're probably better off using cloud hosting and focusing on your unique offering than having that expertise and coordination in house.
Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#319I am not sure what is the use case for this. I would love to see the following projects instead: - exposing a transactional API for S3 - transactional filesystem
I think we’re moving in that direction. I’m really interested to do more in the API space than traditional storage has allowed. Tell me a bit more what you mean by “transactional file system”?
Curently when we use filesystems we actually rely on kernel functionality to have persistence. Relying on syncing[1] can introduce interesting bugs. I can imagine a scenario where the FS has an API that is actually transactional and we can use that to transactionally mutate the content of files.
instead of relying on fsync.
1. fsync, fdatasync - synchronize a file's in-core state with
storage deviceRe: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS
#320Is this like JuiceFS? https://juicefs.com/
After reading the document, I think "proprietary block format" is a design. Obviously this design can provide parallel data flush back to S3 quickly, and we can cache the reading block and prefetch the next block for more cache friendly.