Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

11–20 of 456 posts

Re: S3 is files, but not a filesystem

#11
post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

Weird that it says folders now. I remember it being very strictly called a prefix when I was at AWS.

I think it's just the Web console, It's still prefix in the APIs and CLI.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

Re: S3 is files, but not a filesystem

#13
post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

What exactly do you think a folder is? It’s just an abstraction for organising data.

I'm having a lot of fun imagining this being said to a kid who's trying to buy some folders for school.

Re: S3 is files, but not a filesystem

#14

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.

Re: S3 is files, but not a filesystem

#15
post #9

Earlier quoted context omitted.

Is that really so different from how folders work on other systems? A directory inode is just an inode.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

Except, S3 does let you query by prefix and so the keys have more structure than the second diagram implies: they’re not just random keys, the API implies that common prefixes indicate related objects.

Re: S3 is files, but not a filesystem

#16
post #9

Earlier quoted context omitted.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

Except, S3 does let you query by prefix and so the keys have more structure than the second diagram implies: they’re not just random keys, the API implies that common prefixes indicate related objects.

That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories.

> the API implies that common prefixes indicate related objects.

That’s something users do. The API doesn’t imply anything is related.

And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like a directory!) or you can query using /so as a prefix, or /some/dir/fil as a prefix. It’s just a string. It only looks like a directory when you, the user, decide to interpret the / in the file key as a directory separator. You could just as easily use any other character.

Re: S3 is files, but not a filesystem

#18
post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

What exactly do you think a folder is? It’s just an abstraction for organising data.

S3 doesn’t have that abstraction.

The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.

Re: S3 is files, but not a filesystem

#19

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.

Fun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface:

  type IBackend interface {
    Ls(path string) ([]os.FileInfo, error)
    Cat(path string) (io.ReadCloser, error)
    Mkdir(path string) error
    Rm(path string) error
    Mv(from string, to string) error
    Save(path string, file io.Reader) error
    Touch(path string) error
  }
My mum really couldn't care less about the posix semantic as soon as she can see the pictures of my kid which happen to be on S3

[1] https://github.com/mickael-kerjean/filestash

Re: S3 is files, but not a filesystem

#20
post #9

Earlier quoted context omitted.

Is that really so different from how folders work on other systems? A directory inode is just an inode.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder.

Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

Post reply on HN