Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

91–100 of 456 posts

Re: S3 is files, but not a filesystem

#91

Post by Hakkin »

nickcw wrote: post #75

Great article - would have been useful to read before starting out on the journey of making rclone mount (mount your cloud storage via fuse)! After a lot of iterating we eventually came up with the VFS layer in rclone which adapts S3 (or any other similar storage system like Google Cloud Storage, Azure Blob, Openstack Swift, Oracle Object Storage, etc) into a POSIX-ish file system layer in rclone. The actual rclone m…

> If I could change one thing about S3's API I would like an option to read the metadata with the listings. Rclone stores modification times of files as metadata on the object and there isn't a bulk way of reading these, you have to HEAD the object. Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.

I wonder if you couldn't hack this in by storing the metadata in the key name itself? Obviously with the key length limit of 1024 you would be limited in how much metadata you could store, but it's still quite a lot of space, even taking into account the file path. You could use a deliminator that would be invalid in a normalized path, like '//', for example: /path/to/file.txt//mtime=1710066090

You would still be able to fetch "directories" via prefixes and direct files by using '//' as the prefix.

This kind of formatting would probably make it pretty incompatible with other software though.

Re: S3 is files, but not a filesystem

#92

Post by _flux »

dchest wrote: post #59
inkyoto wrote: post #44

Earlier quoted context omitted.

> […] what the special 0-byte object refers to. It represents an empty folder. Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object. You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag , and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket. Deleting «folder/» in a traditional file system, on t…

It's a matter of a client UI implementation. You can't delete a non-empty folder with POSIX API on common filesystems or FTP too. However, there are file managers, FTP clients, and S3 clients that will do that for you by deleting individual files.

But if the S3 semantics are not helping you, e.g. with multiple clients doing copy/move/delete operations in the hierarchy you could still end up with files that are not in "directories".

So essentially an S3 file manager must be able to handle the situation where there are files without a "directory"—and that I assume is also the most common case as well for S3. Might just not have the "directories" in the first place.

Re: S3 is files, but not a filesystem

#93

Post by pepa65 »

hiyer wrote: post #24
klodolph wrote: post #16

Earlier quoted context omitted.

That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…

One operation where this difference is significant is renaming a "folder". In UNIX (and even UNIX-y distributed filesystems like HDFS) a rename operation at "folder" level is O(1) as it only involves metadata changes. In S3, renaming a "folder" is O(number of files).

From reading the above, if you have a folder 'dir' and a file 'dir/file', after renaming 'dir' to 'folder', you would just have 'folder' and 'dir/file'.

Re: S3 is files, but not a filesystem

#95

Post by quickthrower2 »

afiori wrote: post #80
tuwtuwtuwtuw wrote: post #42

Earlier quoted context omitted.

"filesystem" is not a name reserved for Unix-style file systems. There are many types of file system which is not built on according to your description. When I was a kid, I used systems which didn't support directories, but it was still file systems. It's an incorrect take that a system to manage files must follow a set of patterns like the ones you mentioned to be called "file system".

Terms evolve and now filesytem and "system of files" mean different things, I would argue that not supporting folders or many other file operations make something not a filesystem today.

Yeah hacker used to not mean someone hacking into a computer and breaking a password, then it did then now it means both that and a tech tinkerer.

Re: S3 is files, but not a filesystem

#96

Post by ForHackernews »

vbezhenar wrote: post #71
School-Cotton wrote: post #62

Earlier quoted context omitted.

Amazon doesn’t market S3 as a replacement for file systems, that’s why EBS exists. Also, is S3 really “very expensive”? Relative to what?

S3 usually is the cheapest storage, not only for Amazon, but for other clouds. I don’t understand why.

This is not true in my experience https://www.backblaze.com/cloud-storage/pricing

Re: S3 is files, but not a filesystem

#97

Post by [deleted] »

zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

[deleted]

Re: S3 is files, but not a filesystem

#98

Post by [deleted] »

leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

[deleted]

Re: S3 is files, but not a filesystem

#100

Post by [deleted] »

leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

[deleted]
Post reply on HN