Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

81–90 of 456 posts

Re: S3 is files, but not a filesystem

#81

Post by highwaylights »

leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

This!

I’m fine with it, I actually appreciate the logic and simplicity behind it, but the amount of times I’ve tried to explain why “folders” on S3 keep disappearing while people stare at me like I’m an idiot is really frustrating.

(When you remove the last file in a “folder” on S3, the “folder” disappears, because that pattern no longer appears in the bucket k/v dictionary so there’s no reason to show it as it never existed in the first place).

Re: S3 is files, but not a filesystem

#83

Post by afiori »

zX41ZdbW wrote: post #38
YouWhy wrote: post #23

The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…

ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why…

Is a "write if not exists" atomic operation enouhg as a concurrency primitive for database locks?

Re: S3 is files, but not a filesystem

#85

Post by wkat4242 »

leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

Hmm well there's no folders but if you interact with the object the URL does become nested. So in a sense it does behave exactly like a folder for all intents and purposes when dealing with it that way. It depends what API you use I guess.

I use S3 just as a web bucket of files (I know it's not the best way to do that but it's what I could easily obtain through our company's processes). But in this case it makes a lot of sense though I try to avoid making folders. But other people using the same hosting do use them.

Re: S3 is files, but not a filesystem

#87

Post by inkyoto »

zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

S3 is an object storage, not a file system. The file system in AWS is called EFS. S3 is not positioned as a substitute for file systems, either.

Re: S3 is files, but not a filesystem

#88

Post by raverbashing »

wkat4242 wrote: post #85
leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

Hmm well there's no folders but if you interact with the object the URL does become nested. So in a sense it does behave exactly like a folder for all intents and purposes when dealing with it that way. It depends what API you use I guess. I use S3 just as a web bucket of files (I know it's not the best way to do that but it's what I could easily obtain through our company's processes). But in this case it makes a lo…

Except stuff like s3 cli has all these weird names for normal filesystem items and you have to bang your head to try to figure it out what it all means

(also don't get me started on the whole s3api thing)

Re: S3 is files, but not a filesystem

#89

Post by cwillu »

inkyoto wrote: post #44
riehwvfbk wrote: post #20

Earlier quoted context omitted.

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

> […] what the special 0-byte object refers to. It represents an empty folder. Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object. You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag , and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket. Deleting «folder/» in a traditional file system, on t…

Deleting folder/ in a traditional file system will _fail_ if the folder is not empty. Userspace needs to recurse over the directory structure to unlink everything in it before unlinking the actual folder.

Re: S3 is files, but not a filesystem

#90

Post by okr »

hiyer wrote: post #24
klodolph wrote: post #16

Earlier quoted context omitted.

That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…

One operation where this difference is significant is renaming a "folder". In UNIX (and even UNIX-y distributed filesystems like HDFS) a rename operation at "folder" level is O(1) as it only involves metadata changes. In S3, renaming a "folder" is O(number of files).

Imho, renaming "folders" on S3 results in copying and deleting O(number of files)
Post reply on HN