Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

121–130 of 456 posts

Re: S3 is files, but not a filesystem

#121

Post by jacobsimon »

orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

What is it about S3 that enables this speed, and why can’t traditional Unix file systems do the same?

Re: S3 is files, but not a filesystem

#122

Post by orf »

adrian_b wrote: post #119
orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

Since 30 years ago (starting with XFS in 1993, which was inspired by HPFS) all the good UNIX file systems implement the directories as some kind of B trees. Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.

Yes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents.

You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX.

Regardless of if you can do it with a single directory, you can’t do it for all files recursively under a given prefix. You can’t just ignore directories, or say that “for this list request, ‘-‘ is my directory separator”.

The use of b-trees in file systems is completely beside the point.

Re: S3 is files, but not a filesystem

#123

Post by adrian_b »

orf wrote: post #122
adrian_b wrote: post #119

Earlier quoted context omitted.

Since 30 years ago (starting with XFS in 1993, which was inspired by HPFS) all the good UNIX file systems implement the directories as some kind of B trees. Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.

Yes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents. You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX. Regardless of if you can do it with a single directory, you can’t do it for all files…

The POSIX API is indeed even older, so it is not helpful.

But as you say, there are filesystem-specific methods or operating-system specific methods to reach the true performance of the filesystem.

It is likely that for maximum performance one would have to write custom directory search functions using directly the Linux syscalls, instead of using the standard libc functions, but I would rather do that instead of paying for S3 or something like it.

Re: S3 is files, but not a filesystem

#124

Post by orf »

jacobsimon wrote: post #121
orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

What is it about S3 that enables this speed, and why can’t traditional Unix file systems do the same?

S3 doesn’t have directories, it could be thought of a flat + sorted list of keys.

UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories.

So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Re: S3 is files, but not a filesystem

#125

Post by orf »

adrian_b wrote: post #123
orf wrote: post #122

Earlier quoted context omitted.

Yes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents. You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX. Regardless of if you can do it with a single directory, you can’t do it for all files…

The POSIX API is indeed even older, so it is not helpful. But as you say, there are filesystem-specific methods or operating-system specific methods to reach the true performance of the filesystem. It is likely that for maximum performance one would have to write custom directory search functions using directly the Linux syscalls, instead of using the standard libc functions, but I would rather do that instead of pay…

Yes. You could also just use a SQLite table with two columns (path, contents), then just query that. Or do any number of other things.

The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…

Re: S3 is files, but not a filesystem

#126

Post by Lucasoato »

tison wrote: post #109

It's ever discussed in https://github.com/apache/arrow-rs/issues/3888 for comparing object_store in Apache Arrow to the APIs provided by Apache OpenDAL. Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage. A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage. Other solut…

I'm not sure if Alluxio could be substituted by OpenDAL as a local cache layer for TrinoDB.

Re: S3 is files, but not a filesystem

#127

Post by aeyes »

orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

And if for some reason you need a complete listing along with object sizes and other attributes you can get one every 24 hours with S3 inventory report.

That has always been good enough for me.

Re: S3 is files, but not a filesystem

#128

Post by tjoff »

orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

Is listing really such a key feature that people use it as a database to find objects?

Have not used S3, but that is not how I imagined using it.

Re: S3 is files, but not a filesystem

#129

Post by anamexis »

orf wrote: post #125
adrian_b wrote: post #123

Earlier quoted context omitted.

The POSIX API is indeed even older, so it is not helpful. But as you say, there are filesystem-specific methods or operating-system specific methods to reach the true performance of the filesystem. It is likely that for maximum performance one would have to write custom directory search functions using directly the Linux syscalls, instead of using the standard libc functions, but I would rather do that instead of pay…

Yes. You could also just use a SQLite table with two columns (path, contents), then just query that. Or do any number of other things. The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…

> The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…

Where did this goalpost come from? S3 is not portable or POSIX compliant.

Re: S3 is files, but not a filesystem

#130

Post by bradleyjg »

orf wrote: post #124
jacobsimon wrote: post #121

Earlier quoted context omitted.

What is it about S3 that enables this speed, and why can’t traditional Unix file systems do the same?

S3 doesn’t have directories, it could be thought of a flat + sorted list of keys. UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories. So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Code written against s3 is not portable either. It doesn’t support azure or gcp, much less some random proprietary cloud.
Post reply on HN