Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

271–280 of 456 posts

Re: S3 is files, but not a filesystem

#271

Post by catlifeonmars »

anonymous-panda wrote: post #259
catlifeonmars wrote: post #235

Earlier quoted context omitted.

S3 is fundamentally a key value store. The fact that you can view objects in “directories” is nothing more than a prefix filter. It is not a file system and has no concept of directories.

Directories make up a hierarchical filesystem, but it’s not a necessary condition. A filesystem at its core is just a way of organizing files. If you’re storing and organizing files in s3 then it’s a filesystem for you. Saying it’s “fundamentally a key value store” like it’s something different is confusing because a filesystem is just a key value store of path to contents of file. Indeed there’s every reason to beli…

In context of the comment I was addressing, it’s clear that filesystem means more than just a key value store. I’d argue that this is generally true in common vernacular.

Re: S3 is files, but not a filesystem

#272

Post by orf »

tjoff wrote: post #166
orf wrote: post #158

Earlier quoted context omitted.

My overall point can be summarised as this: - Listing things is a very common operation to do. - The POSIX api and the directory/file hierarchy it provides is a restrictive one. - S3 does not suffer from this, you can recursively list and group keys into directories at “list time”. - If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesys…

Listing things is very common, so why would you outsource that to S3 when all your bookkeeping is elsewhere? It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. For sure, for maintenance tasks etc. it sounds quite useful. And good hygiene with prefixes sounds like a sane idea. But listing being a critical part of what "makes S3 useful"? That…

> It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem.

Because there is no POSIX api for this. Depending on your requirements and query patterns, you may not need a completely separate database that you need to keep in sync.

Re: S3 is files, but not a filesystem

#273

Post by Helmut10001 »

senderista wrote: post #262
Helmut10001 wrote: post #208

Earlier quoted context omitted.

Agree. > S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc). I would expect any cloud provider to be able to detect bitrot these days.

I think the point the OP was trying to make is that they regularly detected bitrot due to their scale, not that they were merely capable of doing so.

Ah, thank you. This makes more sense. And I think I remember reading about it once. Apologies for the misinterpretation!

Re: S3 is files, but not a filesystem

#274

Post by throwitaway222 »

dathery wrote: post #47
throwitaway222 wrote: post #41

Earlier quoted context omitted.

That's just an implementation detail of well known filesystems.

Yes, which is why it's not ideal to reuse the folder metaphor here. Users have an idea how directories work on well-known filesystems and get confused when these fake folders don't behave the same way.

Are all your s3 keys opaque strings (like UUIDs)?, do you use / (slash) in your keys?

If you truly believe S3 has absolutely no connection to folders, you would answer Yes and No.

Re: S3 is files, but not a filesystem

#275

Post by spintin »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

Correct me if I'm wrong but bitrot only affects spinning rust since NAND uses ECC?

If you see this I wonder if S3 is planning on adding hardlinks?

Re: S3 is files, but not a filesystem

#276

Post by aseipp »

BirAdam wrote: post #232

Underneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.

No there isn't. AWS does not use the traditional filesystem layer to store data; that would be a massive mistake from a performance and reliability POV; the POSIX filesystem specification is notoriously vague about things like fsync consistency under particular scenarios, i.e. "do I need to fsync the parent directory before or after fsyncing the contents" for instance and has many bizarre performance cliffs if you aren't careful. At the scale AWS is at even a 10% performance cliff or performance delta would be worth clawing back if it meant removing the POSIX filesystem.

Filesystems are not free; they incur "complexity" (that favorite bugbear everyone on HN loves to complain about) just as much as any other component in the stack does.

> If you stand up an S3 instance with Ceph,

Okay, but AWS does not run on Ceph. Even then, Ceph is an example that recommends the opposite. Nowadays they recommend solutions like the Bluestore OSD backend to store actual data directly on raw block devices, completely bypassing the filesystem layer -- for the exact same reasons I outlined above and many, many others (the actual metadata does use "BlueFS" which is a small FS shim, but this is mostly so that RocksDB can write directly to the block device too, next to the data segments, and BlueFS is in no way a real POSIX filesystem, it's just a shim for existing software).

See "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" written by the Ceph authors[1] about why they finally gave in and wrote Bluestore. The spoiler alert is they got rid of the filesystem precisely because "a filesystem with files" underneath, as you describe, was problematic and worked poorly in comparison (see the conclusion in Section 9.)

Many places do use POSIX filesystems for various reasons, even at large scale, of course.

[1] https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

Re: S3 is files, but not a filesystem

#277

Post by rootbeerdan »

overstay8930 wrote: post #219
cynicalsecurity wrote: post #110

Backblaze B2 is worth mentioning while we are speaking of S3. I'm absolutely in love with their prices (3 times lower than of S3). (I'm not their representative).

We liked B2 but not enough to pay for IPv4 addresses, insane they advertise as a multi-cloud solution but basically kill any chance at adoption when NAT gateways and IPv4 charges are everywhere. We would literally save money paying B2 bandwidth fees (high read low write) but not when being pushed through a NAT64 gateway, or paying an hourly charge just to be able to access B2.

[dead]

Re: S3 is files, but not a filesystem

#278

Post by sgtnoodle »

spintin wrote: post #275
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

Correct me if I'm wrong but bitrot only affects spinning rust since NAND uses ECC? If you see this I wonder if S3 is planning on adding hardlinks?

Pretty much any modern storage medium depends on a healthy amount of error correcting code.

Re: S3 is files, but not a filesystem

#279

Post by mannyv »

rsync wrote: post #206
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

How does the latest ZFS bug impact your bitrot statement?

I mean, technically it’s not bitrot if zeros were accidentally written out instead of data.

Re: S3 is files, but not a filesystem

#280

Post by Veserv »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

But they asked if the claims were audited by a unbiased third party. Are there such audits?

Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

Post reply on HN