101–110 of 456 posts
I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.
The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…
Earlier quoted context omitted.
"filesystem" is not a name reserved for Unix-style file systems. There are many types of file system which is not built on according to your description. When I was a kid, I used systems which didn't support directories, but it was still file systems. It's an incorrect take that a system to manage files must follow a set of patterns like the ones you mentioned to be called "file system".
Terms evolve and now filesytem and "system of files" mean different things, I would argue that not supporting folders or many other file operations make something not a filesystem today.
I could create a system not supporting folders because it relies on tags or something else. Or I could create a system which is write-only and doesn't support rename or delete.
These systems would be file systems according to how the term has been used for 40 (?) years at least. Just don't see any point in restricting the term to exclude random variants.
Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.
Is there a generic name for these distributed cloud file storages? AWS is S3, google is buckets, Azure is blob storage, the open source version is … ?
Object Storage
Earlier quoted context omitted.
That’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like…
One operation where this difference is significant is renaming a "folder". In UNIX (and even UNIX-y distributed filesystems like HDFS) a rename operation at "folder" level is O(1) as it only involves metadata changes. In S3, renaming a "folder" is O(number of files).
More like O(max(number of files, total file size)). You can’t rename objects in S3. To simulate a rename, you have to copy an object and then delete the old one.
Unlike renames in typical file systems, that isn’t atomic (there will be a time period in which both the old and the new object exist), and it becomes slower the larger the file.
Great article - would have been useful to read before starting out on the journey of making rclone mount (mount your cloud storage via fuse)! After a lot of iterating we eventually came up with the VFS layer in rclone which adapts S3 (or any other similar storage system like Google Cloud Storage, Azure Blob, Openstack Swift, Oracle Object Storage, etc) into a POSIX-ish file system layer in rclone. The actual rclone m…
Agree. In MinIO (disclaimer: I work there) we added a "secret" parameter (metadata=true) to include metadata and tags in listings if the user has the appropriate permissions. Of course it being an extension it is not really something that you can reliably use. But rclone can of course always try it and use it if available :)
> You can create zero length files ending in /
Yeah. Though you could also consider "shared prefixes" in listings as directories by itself. That of course makes directories "stateless" and unable to exist if there are no objects in there - which has pros and cons.
> Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.
Yes, that gives severe limitations to clients. However it does make the "server" time the reference. But we have to deal with the same limitation for client side replication/mirroring.
My personal biggest complaint is that there isn't a `HeadObjectVersions` that returns version information for a single object. `ListObjectVersions` is always going to be a "cluster-wide" operation, since you cannot know if the given prefix is actually a prefix or an object key. AWS recently added "GetObjectAttributes" - but it doesn't add version information, which would have fit in nicely there.
The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…
Whether something is or isn't a filesystem requires defining what that actually is. A system that stores files would be a simple explanation. Which is clearly something S3 is capable of. This probably upsets the definition gatekeepers for whatever more specific definitions they are guarding. But it has a nice simple logic to it.
It's worth considering that file systems have had a long history, weren't always the way they are now, and predate the invention of relational databases (like postgres). Technically before hard disks were invented in the fifties, we had no file systems. Just tapes and punch cards. A tape would consist a single blob of bits, which you'd load in memory. Or it would have multiple such blobs at known offsets. I had cassettes full of games for my commodore 64. But no disk drive. These blobs were called files but there was no file system. Sometime, after the invention of disks file systems were invented in the early sixties.
Hierarchical databases were common before relational databases and filesystems with directories are basically a hierarchical database. S3 lacking hierarchy as a simpler key value store clearly isn't a hierarchical database. But of course it's easy to mimic one simply by using / characters in the keys. Which is how the fuse driver probably fakes directories. And S3 even has APIs to listfiles with a common prefix. A bigger deal is the inability to modify files. You can only replace them with other files (delete and add). That kind of is a show stopper for a database. Replacing the entire database on every write isn't very practical.
Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage.
A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage.
Other solutions exist to manage filesystem-like interfaces over S3, including Alluxio and JuiceFS. Unlike Apache OpenDAL, Alluxio and JuiceFS need to be deployed standalone and have a dedicated internal metadata service.