Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

31–40 of 456 posts

Re: S3 is files, but not a filesystem

#31

Post by halayli »

leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

I don't know why you are being downvoted, what you said is true and confuses many newcomers.

Re: S3 is files, but not a filesystem

#32

Post by keithalewis »

klodolph wrote: post #9
riehwvfbk wrote: post #5

Earlier quoted context omitted.

Is that really so different from how folders work on other systems? A directory inode is just an inode.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

Even youngsters are yelling at clouds now. Just a different kind of cloud.

Re: S3 is files, but not a filesystem

#33

Post by fiddlerwoaroof »

klodolph wrote: post #16
fiddlerwoaroof wrote: post #15

Earlier quoted context omitted.

Except, S3 does let you query by prefix and so the keys have more structure than the second diagram implies: they’re not just random keys, the API implies that common prefixes indicate related objects.

That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…

> That’s something users do. The API doesn’t imply anything is related.

Querying ids by prefix doesn’t make any sense for a normal ID type. Just making this operation available and part of your public API indicates that prefixes are semantically relevant to your API’s ID type.

Re: S3 is files, but not a filesystem

#34

Post by klodolph »

3weeksearlier wrote: post #28
klodolph wrote: post #18

Earlier quoted context omitted.

S3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.

It sounds like they have that abstraction in the UI. But if the CLI and API don't have it too, that's weird.

Yeah, the UI and CLI show you “folders”. It’s a client-side thing that doesn’t exist in the actual service. Behind the scenes, the clients are making specific types of queries on the object keys.

You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).

Re: S3 is files, but not a filesystem

#35

Post by klodolph »

fiddlerwoaroof wrote: post #33
klodolph wrote: post #16

Earlier quoted context omitted.

That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…

> That’s something users do. The API doesn’t imply anything is related. Querying ids by prefix doesn’t make any sense for a normal ID type. Just making this operation available and part of your public API indicates that prefixes are semantically relevant to your API’s ID type.

“Prefix” is not the same thing as “directory”.

I can look up names with the prefix “B” and get Bart, Bella, Brooke, Blake, etc. That doesn’t imply that there’s some kind of semantics associated with prefixes. It’s just a feature of your system that you may find useful. The fact that these names have a common prefix, “B”, is not a particularly interesting thing to me. Just like if I had a list of files, 1.jpg, 10.jpg, 100.jpg, it’s probably not significant that they’re being returned sequentially (because I probably want 2.jpg after 1.jpg).

Re: S3 is files, but not a filesystem

#38

Post by zX41ZdbW »

YouWhy wrote: post #23

The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…

ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why external, strongly consistent metadata storage is needed, which is handled by ClickHouse Keeper.

Re: S3 is files, but not a filesystem

#40

Post by throwitaway222 »

klodolph wrote: post #18
solumunus wrote: post #6

Earlier quoted context omitted.

What exactly do you think a folder is? It’s just an abstraction for organising data.

S3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.

Similarly the UI in linux is making up the notion of folders and files in them. But we don't say it doesn't exist.
Post reply on HN