Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

news.ycombinator.com

Post reply on HN
Print view View original

261–270 of 329 posts

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#261

Post by huntaub »

juancampa wrote: post #258
huntaub wrote: post #89

Earlier quoted context omitted.

Hey -- I think this is something that's in-scope for our custom protocol that we're working on. I'd love to chat more about your needs to make sure that we build something that will work great for you. Would you mind shooting an email to hleath [at] regattastorage.com and we can chat more?

We're also interested in SQLite shared by multiple processes on something like Regatta but my concerns are the issues described in the SQLite documentation about NFS [1]. Notably "SQLite relies on exclusive locks for write operations, and those have been known to operate incorrectly for some network filesystems." [1] https://sqlite.org/useovernet.html

Ah, yes — there are some specific file locking concerns with NFSv3 (notably that locks aren’t built as leases like in NFSv4). Let me do a double click here, but I know we will be able to support locks correctly with our custom protocol when we launch it by the end of the year.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#262

Post by huntaub »

unit149 wrote: post #245

Was taking a look at pricing features - melted down, paying per month doesn't seem like a bad option; still, the API features 1 hour SLA support for enterprise tier subscribers. S3 bucket systems for cloud hosting services are typically encrypted through AES-256. SSE-S3 or SSE-KMS are available upon request. [1]: https://aws.amazon.com/blogs/aws/new-amazon-s3-encryption-se... Having the API hosted on Regatta's server…

All data cached in Regatta is also encrypted with AES-256

Re: bring your own compute: It’s certainly something we’re thinking about. We are in discussions with a lot of customers running GPU clusters with orphaned NVMe resources that they would like to install Regatta on. We’d love to get more details on who’s out there looking for this, so please shoot me an email at hleath [at] regattastorage.com

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#263

Post by huntaub »

shaklee3 wrote: post #229
lijok wrote: post #162

Earlier quoted context omitted.

That would sell like hot cakes to the public sector.

The public sector is typically air-gapped, so not really.

I think it depends which part of the public sector! AWS GovCloud is not airgapped, but I certainly know of deployments which are.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#264

Post by huntaub »

datadeft wrote: post #254
hades32 wrote: post #239

Earlier quoted context omitted.

Wouldn't that limit the concurrency of the lambdas to 1? Since they would hold a lock on the db file

Well these are the details that many of us is interested.

This is super dependent on the application, and not something that I could answer without being an expert in SQLite. If SQLite only allows a single reader or writer, then yes. This could still be a good choice for applications which elect a “leader” to serve the database, though.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#265

Post by huntaub »

datadeft wrote: post #250

I am not sure what is the use case for this. I would love to see the following projects instead: - exposing a transactional API for S3 - transactional filesystem

I think we’re moving in that direction. I’m really interested to do more in the API space than traditional storage has allowed. Tell me a bit more what you mean by “transactional file system”?

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#266

Post by huntaub »

mbrt wrote: post #238
huntaub wrote: post #236

Earlier quoted context omitted.

Without getting too much into internals which could change at any time, yes. You have to replicate, partition, and serve consensus over data to achieve high-durability and availability. For concurrent updates, the standard practice for remote file systems is to use file locking to coordinate concurrent writes. Otherwise, NFS doesn't have any guarantees about WRITE operation ordering. If you're talking about concurren…

Thanks for the details! Consistency at the individual file can be guaranteed this way, but I don't think this works across multiple files (as you need a global total order of operations). In any case, this is a pragmatic solution, and I like the tradeoffs. Comparing against NFS rather than Spanner seems the right way to look at it.

This is actually also interesting, in that I don’t think that the file system paradigm actually requires a global total ordering of operations (and, in fact, many file systems don’t provide this). I know that sounds like snapshots wouldn’t be valid, but I think that applications which really care about data consistency (such as databases) are built specifically to handle this (with things like write-ahead-logs).

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#267

Post by huntaub »

hades32 wrote: post #240
huntaub wrote: post #172

Earlier quoted context omitted.

Today, we only charge for cache usage (storage) and data transfer between Regatta and S3. If your metadata access doesn't require transfer to S3, then it doesn't cost anything! However, renames do require transfer to S3 (because we have to move the object on the backend).

does that mean you pay for the storage twice (i.e. S3 and Regatta) or is the cache size tunable?

That’s correct — you pay for the storage yourself in S3, and then you pay for the storage when it’s in the Regatta cache. We may expose the ability to limit the cache size in the future for teams who need controllable costs more than the highest performance.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#269

Post by huntaub »

imcritic wrote: post #246

Regatta Storage is a new cloud file system^W service

Well, I think this is the benefit that our customers are looking for. They aren’t interested in becoming storage administrators, and running Regatta as a service allows them to not. There are, of course, other teams who do want to do that. It’s great that both kinds of products can exist.

Re: Launch HN: Regatta Storage (YC F24) – Turn S3 into a local-like, POSIX cloud FS

#270

Post by kingnothing »

wongarsu wrote: post #185
nine_k wrote: post #155

Earlier quoted context omitted.

Hiring someone who knows how to manage bare metal (with failover and stuff) may take time %)

You pay a datacenter to put it in a rack and add connect power and uplinks, then treat it like a big ec2 instance (minus the built-in firewall). Now you just need someone who knows how to secure an ec2 instance and run your preferred software there (with failover and stuff). If you run a single-digit number of servers and replace them every 5 years you will probably never get a hardware failure. If you're unlucky and…

Are you going to risk your entire business over "probably never get a hardware failure" that, if it hits, might result in days of downtime to resolve? I wouldn't.
Post reply on HN