Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

71–80 of 456 posts

Re: S3 is files, but not a filesystem

#71

Post by vbezhenar »

School-Cotton wrote: post #62
zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

Amazon doesn’t market S3 as a replacement for file systems, that’s why EBS exists. Also, is S3 really “very expensive”? Relative to what?

S3 usually is the cheapest storage, not only for Amazon, but for other clouds. I don’t understand why.

Re: S3 is files, but not a filesystem

#72

Post by 8n4vidtmkvmk »

hn72774 wrote: post #55

> Filesystem software, especially databases, can't be ported to Amazon S3 Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it. Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.

I still don't understand why you'd want to do it in the first place. Just by some contiguous storage.

Re: S3 is files, but not a filesystem

#73

Post by lukeh »

ithkuil wrote: post #61
riehwvfbk wrote: post #5

Earlier quoted context omitted.

Is that really so different from how folders work on other systems? A directory inode is just an inode.

In S3 each file is identified with a full path. Not only you cannot rename a single file, but you also cannot rename a "folder" (because that would imply a bulk rename on a large number of children of that "folder") This is the fundamental difference between a first class folder and just a convention on prefixes of full path names. If you don't allow renames, it doesn't really make sense to have each "folder" store t…

UMich LDAP server, upon which many were based, stored entrys’ hierarchical (distinguished) names with each entry, which I always found a bit weird. AD, eDirectory, and the OpenLDAP HDB backend don’t have this problem.

Re: S3 is files, but not a filesystem

#74

Post by 8n4vidtmkvmk »

YouWhy wrote: post #23

The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…

[flagged]

Re: S3 is files, but not a filesystem

#75

Post by nickcw »

Great article - would have been useful to read before starting out on the journey of making rclone mount (mount your cloud storage via fuse)!

After a lot of iterating we eventually came up with the VFS layer in rclone which adapts S3 (or any other similar storage system like Google Cloud Storage, Azure Blob, Openstack Swift, Oracle Object Storage, etc) into a POSIX-ish file system layer in rclone. The actual rclone mount code is quite a thin layer on top of this.

The VFS layer has various levels of compatibility "off" where it just does directory caching. In this mode, like the article states you can't read and write to a file simultaneously and you can't write to the middle of a file and you can only write files sequentially. Surprisingly quite a lot of things work OK with these limitations. The next level up is "writes" - this supports nearly all the POSIX features that applications want like being able to read and write to the same file at the same time, write to the middle of the file, etc. The cost for that though is a local copy of the file which is uploaded asynchronously when it is closed.

Here are some docs for the VFS caching modes - these mirror the limitations in the article nicely!

https://rclone.org/commands/rclone_mount/#vfs-file-caching

By default S3 doesn't have real directories either. This means you can't have a directory with no files in, and directories don't have valid metadata (like modification time). You can create zero length files ending in / which are known as directory markers and a lot of tools (including rclone) support these. Not being able to have empty directories isn't too much of a problem normally as the VFS layer fakes them and most apps then write something into their empty directories pretty quickly.

So it is really quite a lot of work trying to convert something which looks like S3 into something which looks like a POSIX file system. There is a whole lot of smoke and mirrors behind the scene when things like renaming an open file happens and other nasty corner cases like that.

Rclone's lower level move/sync/copy commands don't bother though and use the S3 API pretty much as-is.

If I could change one thing about S3's API I would like an option to read the metadata with the listings. Rclone stores modification times of files as metadata on the object and there isn't a bulk way of reading these, you have to HEAD the object. Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.

Re: S3 is files, but not a filesystem

#76

Post by afiori »

klodolph wrote: post #26
riehwvfbk wrote: post #20

Earlier quoted context omitted.

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

> Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees. What, exactly, is inefficient about it? Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store. With a filesystem, if you split a string /some/dir/file.j…

I think it is fair to say that S3 (as named files) is not a filesystem and it is inefficient to use it directly as such for common filesystem use cases; the same way that you could say it for a tarball[0].

This does not make S3 a bad storage, just a bad filesystem, not everything needs to be a filesystem.

Arguably is it good that S3 is not a filesystem, as it can be a leaky abstraction eg in git you cannot have two tags name "v2" and "v2/feature-1" as you cannot have both a file and a folder with the same name.

For something more closely related to URLs than filenames forcing a filesystem abstraction is a limitation as "/some/url", "/some/url/", and "/some/url/some-default-name-decided-by-the-webserver" can be different.[1]

[0] where a different tradeoff is that searching a file by name is slower but reading many small files can be faster.

[1] maybe they should be the same, but enforcing it is a bad idea

Re: S3 is files, but not a filesystem

#77

Post by ozim »

zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

These “file systems” are not file systems and I don’t understand why people expect them to be.

Some people are creating tools that make those services easier to synch with file systems but that is not intended use anyway.

Re: S3 is files, but not a filesystem

#78

Post by afiori »

fiddlerwoaroof wrote: post #33
klodolph wrote: post #16

Earlier quoted context omitted.

That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…

> That’s something users do. The API doesn’t imply anything is related. Querying ids by prefix doesn’t make any sense for a normal ID type. Just making this operation available and part of your public API indicates that prefixes are semantically relevant to your API’s ID type.

by this logic the file "foo/bar/" correspond to the filename "f:o:o:/:b:a:r:/" (using a different caracter as separator)

Re: S3 is files, but not a filesystem

#79

Post by karmasimida »

3weeksearlier wrote: post #2

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

Exactly, especially when the concept of filesystem really is defined before the whole internet scale becomes a thing or reality.

Maybe S3 isn't a filesystem according to this definition, but does it really matter to make it one? I doubt it. The Elastic Filesystem is also an AWS product, but you can't really work as one as you have locally, any folder over 20k files basically will timeout if you do a ls. Does it make EFS a filesystem or not?

Re: S3 is files, but not a filesystem

#80

Post by afiori »

tuwtuwtuwtuw wrote: post #42
klodolph wrote: post #9

Earlier quoted context omitted.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

"filesystem" is not a name reserved for Unix-style file systems. There are many types of file system which is not built on according to your description. When I was a kid, I used systems which didn't support directories, but it was still file systems. It's an incorrect take that a system to manage files must follow a set of patterns like the ones you mentioned to be called "file system".

Terms evolve and now filesytem and "system of files" mean different things,

I would argue that not supporting folders or many other file operations make something not a filesystem today.

Post reply on HN