Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

371–380 of 456 posts

Re: S3 is files, but not a filesystem

#371

Post by tomcam »

dmvdoug wrote: post #362
tomcam wrote: post #355

Earlier quoted context omitted.

> if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive. Speak for yourself. Many of us love our audit logs and show them to strangers whatever we can.

I’m picturing you having a slideshow of audit logs that you make guests to your home sit down and watch with you, like the vacation pictures slideshow of old.

Yes but my audit logs are special and everyone just loves them, although they playfully act bored

Re: S3 is files, but not a filesystem

#372

Post by hiyer »

okr wrote: post #90
hiyer wrote: post #24

Earlier quoted context omitted.

One operation where this difference is significant is renaming a "folder". In UNIX (and even UNIX-y distributed filesystems like HDFS) a rename operation at "folder" level is O(1) as it only involves metadata changes. In S3, renaming a "folder" is O(number of files).

Imho, renaming "folders" on S3 results in copying and deleting O(number of files)

Exactly.

Re: S3 is files, but not a filesystem

#373

Post by riehwvfbk »

klodolph wrote: post #26
riehwvfbk wrote: post #20

Earlier quoted context omitted.

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

> Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees. What, exactly, is inefficient about it? Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store. With a filesystem, if you split a string /some/dir/file.j…

Renaming a folder is inefficient.

Re: S3 is files, but not a filesystem

#374

Post by SlightlyLeftPad »

treflop wrote: post #231
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough. The things that you mentioned are standard storage company activities. Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it. Keepin…

The beauty of proprietary systems is that the only information we can ever expect to get about how they’re built is the biased information we get from the builders of those systems.

Re: S3 is files, but not a filesystem

#376

Post by deathanatos »

fijiaarone wrote: post #304
belter wrote: post #156

Earlier quoted context omitted.

No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

So in reality S3 takes about 2 seconds to retrieve a single file, under ideal conditions. 1 second round trip for the request to DynamoDB to get the object key of the file and 1 second round trip to S3 to get the file contents (assuming no CPU cost on the search because you’re getting the key by ID from the DynamoDB in a flat single table store. And that the file has no network IO because it is a trivial number of by…

… S3's response time is nowhere near 2 seconds. (Or even 1 second.) Like a sibling poster says, 50ms is a much more realistic ballpark for TTFB.

Re: S3 is files, but not a filesystem

#377

Post by Veserv »

cyberax wrote: post #336
Veserv wrote: post #280

Earlier quoted context omitted.

But they asked if the claims were audited by a unbiased third party. Are there such audits? Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

> Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently? It's listed prominently in the public docs: https://aws.amazon.com/s3/storage-classes/

I read that page and it does not provide any contractual durability guarantees as far as I can see. It provides "designed for availability" and then contractual availability SLA guarantees. It provides "designed for durability", but presents no contractual durability guarantee as far as I can see.

Given that their lawyers clearly indicate that "designed for availability" is not what they are contractually obligated to provide, only the letter of the SLA does that; "designed for durability" is similarly a marketing statement that does not incur any contractual obligations. Is there some specific statement in that document that I am missing which indicates that data durability is not fully at their convenience?

Re: S3 is files, but not a filesystem

#378

Post by [deleted] »

cyberax wrote: post #336
Veserv wrote: post #280

Earlier quoted context omitted.

But they asked if the claims were audited by a unbiased third party. Are there such audits? Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

> Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently? It's listed prominently in the public docs: https://aws.amazon.com/s3/storage-classes/

[deleted]

Re: S3 is files, but not a filesystem

#379

Post by 3weeksearlier »

klodolph wrote: post #34
3weeksearlier wrote: post #28

Earlier quoted context omitted.

It sounds like they have that abstraction in the UI. But if the CLI and API don't have it too, that's weird.

Yeah, the UI and CLI show you “folders”. It’s a client-side thing that doesn’t exist in the actual service. Behind the scenes, the clients are making specific types of queries on the object keys. You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).

If you can't rename or delete a folder, yeah, I would say folders don't really exist.

Re: S3 is files, but not a filesystem

#380

Post by mr_toad »

macintux wrote: post #190
belter wrote: post #156

Earlier quoted context omitted.

No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

I don't know that you can characterize that as a "standard practice". Maybe it's widespread, but I've not encountered it.

Hive stores metadata in a relational database. So does Snowflake.
Post reply on HN