Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

61–70 of 456 posts

Re: S3 is files, but not a filesystem

#61

Post by ithkuil »

riehwvfbk wrote: post #5
leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

Is that really so different from how folders work on other systems? A directory inode is just an inode.

In S3 each file is identified with a full path.

Not only you cannot rename a single file, but you also cannot rename a "folder" (because that would imply a bulk rename on a large number of children of that "folder")

This is the fundamental difference between a first class folder and just a convention on prefixes of full path names.

If you don't allow renames, it doesn't really make sense to have each "folder" store the list of the children.

You can instead have a giant ordered map (some kind of b-tree) that allows you for efficient lookup and scanning neighbouring nodes.

Re: S3 is files, but not a filesystem

#62

Post by School-Cotton »

zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

Amazon doesn’t market S3 as a replacement for file systems, that’s why EBS exists.

Also, is S3 really “very expensive”? Relative to what?

Re: S3 is files, but not a filesystem

#63

Post by albert_e »

CobrastanJorji wrote: post #48
3weeksearlier wrote: post #2

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

They are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers…

If a customer makes an IT decision as big as running Hadoop or RDBMS with S3 as storage ... but does not consult at least a Associate level AWS Certified architect (who are doke a dozen) for at least one day worth of advice which is probably a couple of hundred dollars at most ...

Can we really blame AWS?

I am sure none of official AWS documentations or examples show such an architecture.

----

Amazon EMR can run Hadoop and use Amazon S3 as storage via EMR FS.

"S3 mountpoints" are a feature specifically for workloads that need to see S3 as a file system.

For block storage workloads there is EBS and EFS and FSx that AWS heavily advertises.

Re: S3 is files, but not a filesystem

#64

Post by throwaway290 »

zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

It's for building things on top. If you want to rename/move/copy data, implement a layer that maps objects to "filenames" or any metadata you like (or use some lib). If you want to write logs, implement append and rotation. But I for example don't and won't need any of that and if it helps keep the API simpler and more reliable then I benefit.

being a conventional filesystem for S3 would be either a very leaky abstraction or completely different product

Re: S3 is files, but not a filesystem

#65

Post by School-Cotton »

klodolph wrote: post #26
riehwvfbk wrote: post #20

Earlier quoted context omitted.

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

> Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees. What, exactly, is inefficient about it? Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store. With a filesystem, if you split a string /some/dir/file.j…

I think what you’re describing is simply not a hierarchical file system. It’s a different thing that supports different operations and, indeed, is better or worse at different operations.

Re: S3 is files, but not a filesystem

#66

Post by hnlmorg »

type_Ben_struct wrote: post #52

Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.

I don’t know a whole lot about LucidLink but Weka basically uses S3 as a dataplane for their own file system.

Re: S3 is files, but not a filesystem

#67

Post by globular-toast »

A filesystem is an abstraction built on a block device. A block device just gives you a massive array of bytes and lets you read/write from them in blocks (e.g. write these 300 bytes at position 273041).

A block device itself is an abstraction built on real hardware. "Write these 300 bytes" really means something like "move needle on platter 2 to position 6... etc"

S3 is just a different abstraction that is also built on raw storage somehow. It's a strictly flat key-object store. That's it. I don't know why people have a problem with this. If you need "filesystem stuff" then implement it in your app, or use a filesystem. You only need to append? Use a database to keep track of the chain of appends and store the chunks in S3. Doesn't work for you? Use something else. Need to "copy"? Make a new reference to the same object in your db. Doesn't work for you? Use something else.

S3 works for a lot of people. Stop trying to make it something else.

And stop trying to change the meaning of super well-established names in your field. A filesystem is described in text books everywhere. S3 is not a filesystem and never claimed to be one.

Oh and please study a bit of operating system design. Just a little bit. It really helps and is great fun too.

Re: S3 is files, but not a filesystem

#68

Post by Cthulhu_ »

zmmmmm wrote: post #56

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't ren…

They're not filesystems though, they're object storage or key/value storage if you will. It's intended to store the log files for long term once they're full.

You can rename / move a file, but it involves copying and deleting the original; I don't understand why they don't have a shortcut for that, but it probably makes sense that the user of the service is aware of the process instead of hiding it.

I'm not sure about the 5GB limit, it's probably documented somewhere as to why that is; possibly, like tweets, having an upper limit helps them optimize things. Anyway there too there's tools, you can do multipart somethings and there's this official blogpost on the subject: https://aws.amazon.com/blogs/storage/copying-objects-greater...

Interesting to note maybe in the context of the post; copy, rename, moving large files, all that could be abstracted away, but that would hide the underlying logic - which might lead to inefficient usage of the service - and worse, make users think it's just a filesystem and use it accordingly, but it's not intended or designed for that use case.

Re: S3 is files, but not a filesystem

#69

Post by DonHopkins »

fiddlerwoaroof wrote: post #14
3weeksearlier wrote: post #2

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.

Can S3 murder your wife like ReiserFS and Reiser4?

https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...

Post reply on HN