Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

331–340 of 456 posts

Re: S3 is files, but not a filesystem

#331

Post by cuno »

type_Ben_struct wrote: post #52

Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.

The problem with these approaches is that the data is scrambled on the backend, so you can't access the files directly from S3 anymore. Instead you need an S3 gateway to convert from scrambled S3 to unscrambled S3. They rely on a separate database to reassemble the pieces back together again.

Re: S3 is files, but not a filesystem

#332

Post by glitchcrab »

nickcw wrote: post #75

Great article - would have been useful to read before starting out on the journey of making rclone mount (mount your cloud storage via fuse)! After a lot of iterating we eventually came up with the VFS layer in rclone which adapts S3 (or any other similar storage system like Google Cloud Storage, Azure Blob, Openstack Swift, Oracle Object Storage, etc) into a POSIX-ish file system layer in rclone. The actual rclone m…

Hey Nick :wave:

Re: S3 is files, but not a filesystem

#333

Post by electroly »

blakesley wrote: post #317
electroly wrote: post #286

Earlier quoted context omitted.

The way that you said "recursively" and spent a lot of time describing "directories" and "levels" worries me. The fastest way to list objects in S3 wouldn't involve recursion at all; you just list all objects under a prefix. If you're using the path delimiter to pretend that S3 keys are a folder structure (they're not) and go "folder by folder", it's going to be way slower. When calling ListObjectsV2, make sure you a…

All these comments saying merely "S3 has no concept of directories" without an explanation (or at least a link to an explanation) are pretty unhelpful, IMO. I dismissed your comment, but then I came upon this later one explaining why: https://news.ycombinator.com/item?id=39660445 After reading that, I now understand your comment.

I appreciate you sharing that point of view. There's a "curse of knowledge" effect with AWS where its card-carrying proponents (myself included) lose perspective on how complex it actually is.

Re: S3 is files, but not a filesystem

#334

Post by OnlyMortal »

It can be a file system.

I’ve written my own FUSE that uses Rabin Chunking and stores the data (and meta) in S3. The C++/AWS SDK FUSE is connected to a Go SMB server that runs locally on my Mac and works with (local) TimeMachine.

I use Wasabi for cost and speed reasons.

Re: S3 is files, but not a filesystem

#335

Post by cyberax »

jftuga wrote: post #326
breckognize wrote: post #266

Earlier quoted context omitted.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and…

If I were to upload a 50kb object to S3 (standard tier), about how many unique physical copies would exist?

At least 3.

Re: S3 is files, but not a filesystem

#336

Post by cyberax »

Veserv wrote: post #280
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

But they asked if the claims were audited by a unbiased third party. Are there such audits? Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

> Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

It's listed prominently in the public docs: https://aws.amazon.com/s3/storage-classes/

Re: S3 is files, but not a filesystem

#337

Post by chupasaurus »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

> and bigger events like natural disasters

Outdated anecdata: I've worked for a company that lost some parts of buckets after the lightning strike incident in 2011, which bumped the paranoia quite a bit. AFAIK same thing couldn't happen for more than a decade.

Re: S3 is files, but not a filesystem

#339

Post by 4death4 »

treflop wrote: post #231
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough. The things that you mentioned are standard storage company activities. Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it. Keepin…

This was a few years ago, but blob storage on GCP had a global outage due to an outage in a single zone. That, among numerous other issues with GCP, lost my confidence entirely. Maybe it’s better now.

Re: S3 is files, but not a filesystem

#340

Post by immibis »

alexchamberlain wrote: post #284
breckognize wrote: post #216

Earlier quoted context omitted.

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver. The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough tha…

> They're far enough apart for good isolation, which provides durability and availability It can't possibly be enough for critical data though, right? I'm guessing a fire in 1 is unlikely to spread to another, but could it affect the availability of another? What about a deliberate attack on the DCs or the utilities supplying the DCs?

Yes, if a terrorist blows up all of the several Amazon DCs holding your data, your data will be lost. This is true no matter how many DCs are holding your data, who owns them, or where they are. You can improve your chances, of course.

There have been region-wide availability outages before. They're pretty rare and make worldwide news media due to how much of the internet they take out. I don't think there's been S3 data loss since they got serious about preventing S3 data loss.

Post reply on HN