Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

41–50 of 456 posts

Re: S3 is files, but not a filesystem

#41

Post by throwitaway222 »

klodolph wrote: post #34
3weeksearlier wrote: post #28

Earlier quoted context omitted.

It sounds like they have that abstraction in the UI. But if the CLI and API don't have it too, that's weird.

Yeah, the UI and CLI show you “folders”. It’s a client-side thing that doesn’t exist in the actual service. Behind the scenes, the clients are making specific types of queries on the object keys. You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).

That's just an implementation detail of well known filesystems.

Re: S3 is files, but not a filesystem

#42

Post by tuwtuwtuwtuw »

klodolph wrote: post #9
riehwvfbk wrote: post #5

Earlier quoted context omitted.

Is that really so different from how folders work on other systems? A directory inode is just an inode.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

"filesystem" is not a name reserved for Unix-style file systems. There are many types of file system which is not built on according to your description. When I was a kid, I used systems which didn't support directories, but it was still file systems.

It's an incorrect take that a system to manage files must follow a set of patterns like the ones you mentioned to be called "file system".

Re: S3 is files, but not a filesystem

#43

Post by Izkata »

paranoidrobot wrote: post #11
nostrebored wrote: post #8

Earlier quoted context omitted.

Weird that it says folders now. I remember it being very strictly called a prefix when I was at AWS.

I think it's just the Web console, It's still prefix in the APIs and CLI. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

The web console even collapses them like folders on slashes, further obfuscating how it actually works. I remember having to explain to coworkers why it was so slow to load a large bucket.

Re: S3 is files, but not a filesystem

#44

Post by inkyoto »

riehwvfbk wrote: post #20
klodolph wrote: post #9

Earlier quoted context omitted.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

> […] what the special 0-byte object refers to. It represents an empty folder.

Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object.

You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag, and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket.

Deleting «folder/» in a traditional file system, on the other hand, will also delete «file1.txt» and «file2.txt» in it.

Re: S3 is files, but not a filesystem

#45

Post by throwaway892238 »

> The "simple" in S3 is a misnomer. S3 is not actually simple. It's deep.

Simple doesn't mean "not deep". It means having the fewest parts needed in order to accomplish your requirements.

If you require a distributed, centralized, replicated, high-availability, high-durability, high-bandwidth, low-latency, strongly-consistent, synchronous, scalable object store with HTTP REST API, you can't get much simpler than S3. Lots of features have been added to AWS S3 over the years, but the basic operation has remained the same.

Re: S3 is files, but not a filesystem

#46

Post by dathery »

throwitaway222 wrote: post #40
klodolph wrote: post #18

Earlier quoted context omitted.

S3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.

Similarly the UI in linux is making up the notion of folders and files in them. But we don't say it doesn't exist.

Directories actually exist on the filesystem, which is why you have to create them before use and they can exist and be empty. They don't exist in S3 and neither of those properties do, either. Similarly, common filesystem operations on directories (like efficiently renaming them, and thus the files under them) are not possible in S3.

Of course it can still be useful to group objects in the S3 UI, but it would probably be better to use some kind of prefix-centric UI rather than reusing the folder metaphor when it doesn't match the paradigm people are used to.

Re: S3 is files, but not a filesystem

#47

Post by dathery »

throwitaway222 wrote: post #41
klodolph wrote: post #34

Earlier quoted context omitted.

Yeah, the UI and CLI show you “folders”. It’s a client-side thing that doesn’t exist in the actual service. Behind the scenes, the clients are making specific types of queries on the object keys. You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).

That's just an implementation detail of well known filesystems.

Yes, which is why it's not ideal to reuse the folder metaphor here. Users have an idea how directories work on well-known filesystems and get confused when these fake folders don't behave the same way.

Re: S3 is files, but not a filesystem

#48

Post by CobrastanJorji »

3weeksearlier wrote: post #2

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

They are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers who are angry that S3 isn't as fast as some real filesystem solution that the customer migrated from because S3 was cheaper.

When people decide that X is a filesystem, they try to use it like it's a local, POSIX filesystem, and that's terrible because it won't be immediately obvious why it's a stupid plan.

Re: S3 is files, but not a filesystem

#49

Post by kelnos »

throwitaway222 wrote: post #40
klodolph wrote: post #18

Earlier quoted context omitted.

S3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.

Similarly the UI in linux is making up the notion of folders and files in them. But we don't say it doesn't exist.

No, they're not made up. A folder (or directory) is a specific type of inode, just a file is.

S3 doesn't have folders. The UI fakes them by creating a 0-byte object (or file, if you will). It's a kludge.

Post reply on HN