S3 is files, but not a filesystem

Post by nh2 » Sun, Mar 10, 2024, 1:18 PM UTC

adrian_b wrote: post #119

orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

Since 30 years ago (starting with XFS in 1993, which was inspired by HPFS) all the good UNIX file systems implement the directories as some kind of B trees. Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.

Post by arcfour » Sun, Mar 10, 2024, 1:27 PM UTC

bradleyjg wrote: post #130

orf wrote: post #124

Earlier quoted context omitted.

S3 doesn’t have directories, it could be thought of a flat + sorted list of keys. UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories. So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Code written against s3 is not portable either. It doesn’t support azure or gcp, much less some random proprietary cloud.

Post by gjvc » Sun, Mar 10, 2024, 1:36 PM UTC

riehwvfbk wrote: post #20

klodolph wrote: post #9

Earlier quoted context omitted.

Yes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | paren…

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

Post by zaphar » Sun, Mar 10, 2024, 1:42 PM UTC

bradleyjg wrote: post #130

orf wrote: post #124

Earlier quoted context omitted.

S3 doesn’t have directories, it could be thought of a flat + sorted list of keys. UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories. So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Code written against s3 is not portable either. It doesn’t support azure or gcp, much less some random proprietary cloud.

Post by klodolph » Sun, Mar 10, 2024, 1:46 PM UTC

throwitaway222 wrote: post #41

klodolph wrote: post #34

Earlier quoted context omitted.

Yeah, the UI and CLI show you “folders”. It’s a client-side thing that doesn’t exist in the actual service. Behind the scenes, the clients are making specific types of queries on the object keys. You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).

That's just an implementation detail of well known filesystems.