Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

151–160 of 456 posts

Re: S3 is files, but not a filesystem

#151

Post by klodolph »

Demiurge wrote: post #132
klodolph wrote: post #29

Earlier quoted context omitted.

I think if you focus too narrowly on the details of the wire protocol, you’ll lose sight of the big picture and the semantics. S3 is not a filesystem because the semantics are different from the kind of semantics we expect from filesystems. You can’t take the high-level API provided by a filesystem, use S3 as the backing storage, and expect to get good performance out of it unless you use a ton of translation. Stuff…

Right, the NFS/CIFS support writing blocks, but S3 basically does HTTP get and post verbs. I would say that these concepts are the defining difference. To call S3 a filesystem is not wrong in abstract, but it’s not different than calling Wordpress a filesystem, or DNS, or anything that stores something for you. Of course, it will be inefficient to implement a block write on top of any of these, that’s because you hav…

I think the blocks are one part of it, and the other part is that S3 doesn’t support renaming or moving objects, and doesn’t have directories (just prefixes). Whenever I’ve seen something with filesystem-like semantics on top of S3, it’s done by using S3 as a storage layer, and building some other kind of view of the storage on top using a separate index.

For example, maybe you have a database mapping file paths to S3 objects. This gives you a separate metadata layer, with S3 as the storage layer for large blocks of data.

Re: S3 is files, but not a filesystem

#152

Post by klodolph »

_flux wrote: post #92
dchest wrote: post #59

Earlier quoted context omitted.

It's a matter of a client UI implementation. You can't delete a non-empty folder with POSIX API on common filesystems or FTP too. However, there are file managers, FTP clients, and S3 clients that will do that for you by deleting individual files.

But if the S3 semantics are not helping you, e.g. with multiple clients doing copy/move/delete operations in the hierarchy you could still end up with files that are not in "directories". So essentially an S3 file manager must be able to handle the situation where there are files without a "directory"—and that I assume is also the most common case as well for S3. Might just not have the "directories" in the first pla…

I have personally never seen the 0-byte files people keep talking about here. In every S3 bucket I’ve ever looked at, the “directories” don’t exist at all. If you have a dir/file1.txt and dir/file2.txt, there is NO such object as dir. Not even a placeholder.

Re: S3 is files, but not a filesystem

#153

Post by orf »

calpaterson wrote: post #137
orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

I have to say that I'm not hugely convinced. I don't really think that being able to pull out the keys before or after a prefix is particularly impressive. That is the basis for database indices going back to the 1970s after all. Perhaps the use-cases you're talking about are very different from mine. That's possible of course. But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn'…

It depends on a few factors. The list objects call hides deleted and noncurrent versions, but it has to skip over them. Grouping prefixes also takes time, if they contain a lot of noncurrent or deleted keys.

A pathological case would be a prefix with 100 million deleted keys, and 1 actual key at the end. Listing the parent prefix takes a long time in this case - I’ve seen it take several minutes.

If your bucket is pretty “normal” and doesn’t have this, or isn’t versioned, then you can do 4-5 thousand list requests a second, at any given key/prefix, in constant time. Or or you can explicitly list object versions (and not skip deleted keys) also in constant time.

It all depends on your data: if you need to list all objects then yeah it’s gonna be slow because you need to paginate through all the objects. But the point is that you don’t have to do that if you don’t want to, unlike a traditional filesystem with a directory hierarchy.

And this enables parallelisation: why list everything sequentially, when you can group the prefixes by some character (i.e “-“), then process each of those prefixes in parallel.

The world is your oyster.

Re: S3 is files, but not a filesystem

#154

Post by orf »

anamexis wrote: post #138
orf wrote: post #134

Earlier quoted context omitted.

From the article we're commenting on, which is comparing the interface of S3 to the POSIX interface. Not any given filesystem + platform specific interface.

The article does not mention POSIX, or anything about listing files, at all.

It mistakenly mentions UNIX whilst referencing the POSIX filesystem API, and I literally quoted where it talks about listing in my original comment.

Re: S3 is files, but not a filesystem

#155

Post by albert_e »

albert_e wrote: post #63
CobrastanJorji wrote: post #48

Earlier quoted context omitted.

They are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers…

If a customer makes an IT decision as big as running Hadoop or RDBMS with S3 as storage ... but does not consult at least a Associate level AWS Certified architect (who are doke a dozen) for at least one day worth of advice which is probably a couple of hundred dollars at most ... Can we really blame AWS? I am sure none of official AWS documentations or examples show such an architecture. ---- Amazon EMR can run Hado…

*dime a dozen

(Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)

Re: S3 is files, but not a filesystem

#156

Post by belter »

tjoff wrote: post #128
orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.

No. The standard practice is to use a DynamoDB table as the index for your objects in S3.

This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

Re: S3 is files, but not a filesystem

#157

Post by albert_e »

albert_e wrote: post #58
hn72774 wrote: post #55

> Filesystem software, especially databases, can't be ported to Amazon S3 Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it. Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.

There is specifically block storage service (EBS) and falvirs of it like EBS multi-attach and EFS that can ne used if there is a need to port software/databases to the cloud with low level filesystem support. Why would we need to do it on object storage which addresses a different type of storage need. Nevertheless there are projects like EMRFS and S3 file system mount points that try to provide files stem interfaces…

*flavors

*can be used

*file system

(Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)

Re: S3 is files, but not a filesystem

#158

Post by orf »

tjoff wrote: post #140
orf wrote: post #133

Earlier quoted context omitted.

Sure. It's kind of an index - limited to prefix-only searching, but useful. Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`. If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get…

But surely you need to track that elsewhere anyway? That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.

My overall point can be summarised as this:

- Listing things is a very common operation to do.

- The POSIX api and the directory/file hierarchy it provides is a restrictive one.

- S3 does not suffer from this, you can recursively list and group keys into directories at “list time”.

- If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesystem, you shouldn’t need to list 1k+ keys sequentially apart from during maintenance tasks.

- This is actually quite fast, compared to alternatives.

Whether or not you see a use case for this is sort of irrelevant: they exist. it’s what allows you to easily put data into s3 and flexibly group/scan it by specific attributes.

Re: S3 is files, but not a filesystem

#160

Post by _flux »

klodolph wrote: post #152
_flux wrote: post #92

Earlier quoted context omitted.

But if the S3 semantics are not helping you, e.g. with multiple clients doing copy/move/delete operations in the hierarchy you could still end up with files that are not in "directories". So essentially an S3 file manager must be able to handle the situation where there are files without a "directory"—and that I assume is also the most common case as well for S3. Might just not have the "directories" in the first pla…

I have personally never seen the 0-byte files people keep talking about here. In every S3 bucket I’ve ever looked at, the “directories” don’t exist at all. If you have a dir/file1.txt and dir/file2.txt, there is NO such object as dir. Not even a placeholder.

Yeah, this post was the first one I had even heard of them.
Post reply on HN