[dead]
S3 is files, but not a filesystem
131–140 of 456 posts
Re: S3 is files, but not a filesystem
#132Earlier quoted context omitted.
Let’s start with the fact that you’re talking to an HTTP api… Even if S3 had web3.0 inodes, the querying semantics would not make sense. It’s a higher level API, because you don’t deal with blocks of magnetic storage and binary buffers. Of course s3 is not a filesystem, that is part of its definition, and reason to be…
I think if you focus too narrowly on the details of the wire protocol, you’ll lose sight of the big picture and the semantics. S3 is not a filesystem because the semantics are different from the kind of semantics we expect from filesystems. You can’t take the high-level API provided by a filesystem, use S3 as the backing storage, and expect to get good performance out of it unless you use a ton of translation. Stuff…
Right, the NFS/CIFS support writing blocks, but S3 basically does HTTP get and post verbs. I would say that these concepts are the defining difference. To call S3 a filesystem is not wrong in abstract, but it’s not different than calling Wordpress a filesystem, or DNS, or anything that stores something for you. Of course, it will be inefficient to implement a block write on top of any of these, that’s because you have to literally do it yourself. As in, download the file, edit it, upload again.
Re: S3 is files, but not a filesystem
#133> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…
Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.
Sure. It's kind of an index - limited to prefix-only searching, but useful.
Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`.
If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get a set of all users who have photos, you'd list `[company-uuid]/` with a Delimiter set to `.`
The point is that it's flexible and with a bit of thought it allows you to "remove all a users uploads between two dates", "remove all a companies uploads" or "remove all a users uploads" with a single call. Or whatever specific stuff is important to your use-case, that might otherwise need a separate DB.
It's not perfect - you can't reverse the listing (i.e you can't get the latest photo for a given user by sorting descending for example), and needs some thought about your key structure.
Re: S3 is files, but not a filesystem
#134Earlier quoted context omitted.
Yes. You could also just use a SQLite table with two columns (path, contents), then just query that. Or do any number of other things. The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…
> The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then… Where did this goalpost come from? S3 is not portable or POSIX compliant.
From the article we're commenting on, which is comparing the interface of S3 to the POSIX interface. Not any given filesystem + platform specific interface.
Re: S3 is files, but not a filesystem
#135Earlier quoted context omitted.
Yes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents. You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX. Regardless of if you can do it with a single directory, you can’t do it for all files…
The POSIX API is indeed even older, so it is not helpful. But as you say, there are filesystem-specific methods or operating-system specific methods to reach the true performance of the filesystem. It is likely that for maximum performance one would have to write custom directory search functions using directly the Linux syscalls, instead of using the standard libc functions, but I would rather do that instead of pay…
There are no specific syscalls that you can use for this. The libc functions and the syscalls are extremely similar.
Re: S3 is files, but not a filesystem
#136Check out kopia.io is a backup software that uses S3 to store files by blocks or pages.
You can browse, search and sort the files and directories of the different snapshot or versions of the file.
I love it !
For me it's a file system in S3.
Bonus: you must use a key, to encrypt the files.
Re: S3 is files, but not a filesystem
#137> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…
I have to say that I'm not hugely convinced. I don't really think that being able to pull out the keys before or after a prefix is particularly impressive. That is the basis for database indices going back to the 1970s after all.
Perhaps the use-cases you're talking about are very different from mine. That's possible of course.
But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn't have to get very big before listing the keys takes longer than reading them. I seem to remember that listing operations ran at sub-1mbps, but admittedly I don't have a big bucket handy right now to test that.
Re: S3 is files, but not a filesystem
#138Earlier quoted context omitted.
> The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then… Where did this goalpost come from? S3 is not portable or POSIX compliant.
From the article we're commenting on, which is comparing the interface of S3 to the POSIX interface. Not any given filesystem + platform specific interface.
The article does not mention POSIX, or anything about listing files, at all.
Re: S3 is files, but not a filesystem
#139Re: S3 is files, but not a filesystem
#140Earlier quoted context omitted.
Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.
Sure. It's kind of an index - limited to prefix-only searching, but useful. Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`. If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get…
But surely you need to track that elsewhere anyway?
That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.