21–30 of 456 posts
My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…
Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I suspect latency would kill the hypothetical use case of Postgres over S3 even if S3 had incorporated the necessary API semantics - could somebody more knowledgeable chime in?).
A more interesting venue to pursue would be - what other additions could be made to the S3 API to make it more usable on its own right - for example, why doesn't S3 offer more than one filename per blob? (e.g., a similar to what links do in POSIX)
Earlier quoted context omitted.
Except, S3 does let you query by prefix and so the keys have more structure than the second diagram implies: they’re not just random keys, the API implies that common prefixes indicate related objects.
That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…
Earlier quoted context omitted.
Is that really so different from how folders work on other systems? A directory inode is just an inode.
Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…
Earlier quoted context omitted.
Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…
Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.
What, exactly, is inefficient about it?
Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store.
With a filesystem, if you split a string /some/dir/file.jpg into three parts, “some”, “dir”, “file.jpg”, then you are actually making a decision about the tree structure. And here’s a question—is that a balanced tree you got there? Maybe it’s completely unbalanced! That’s actually inefficient.
Let’s suppose, instead, you treat the key as a plain string and stick it in a tree. You have a lot of freedom now, in how you balance the tree, since you are not forced to stick nodes in the tree at every / character.
It’s just a different efficiency tradeoff. Certain operations are now much less efficient (like “rename a directory” which, on S3, is actually “copy a zillion objects). Some operations are more efficient, like “store a file” or “retrieve a file”.
Earlier quoted context omitted.
Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.
Fun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } My mum really couldn't care less about…
Earlier quoted context omitted.
What exactly do you think a folder is? It’s just an abstraction for organising data.
S3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.
Earlier quoted context omitted.
Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…
Let’s start with the fact that you’re talking to an HTTP api… Even if S3 had web3.0 inodes, the querying semantics would not make sense. It’s a higher level API, because you don’t deal with blocks of magnetic storage and binary buffers. Of course s3 is not a filesystem, that is part of its definition, and reason to be…
S3 is not a filesystem because the semantics are different from the kind of semantics we expect from filesystems. You can’t take the high-level API provided by a filesystem, use S3 as the backing storage, and expect to get good performance out of it unless you use a ton of translation.
Stuff like NFS or CIFS are filesystems. They behave like filesystems, in practice. You can rename files. You can modify files. You can create directories.
The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…
There's also the OG Aurora whitepaper: https://www.amazon.science/publications/amazon-aurora-design...