Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

201–210 of 456 posts

Re: S3 is files, but not a filesystem

#201

Post by dekhn »

tjoff wrote: post #191
kbolino wrote: post #174

Earlier quoted context omitted.

> But surely you need to track that elsewhere anyway? Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else. Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional servic…

I agree, but something as simple (in functionality) as that ought to be an edge-case. Not a defining feature of S3.

it's a property of the system that I, as an architect, would seriously consider as part of my system's design. I've worked with many systems where iterating over items in order starting from a prefix is extremely cheap (sstables).

Re: S3 is files, but not a filesystem

#202

Post by svat »

It's nice to see Ousterhout's idea of module depth (the main idea from his A Philosophy of Software Design) getting more mainstream — mentioned in this article with attribution only in "Other notes", which suggests the author found it natural enough not to require elaboration. Being obvious-in-hindsight like this is a sign of a good idea. :-)

> The concept of deep vs shallow modules comes from John Ousterhout's excellent book. The book is [effectively] a list of ideas on software design. Some are real hits with me, others not, but well worth reading overall. Praise for making it succinct.

Re: S3 is files, but not a filesystem

#203

Post by orf »

tjoff wrote: post #191
kbolino wrote: post #174

Earlier quoted context omitted.

> But surely you need to track that elsewhere anyway? Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else. Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional servic…

I agree, but something as simple (in functionality) as that ought to be an edge-case. Not a defining feature of S3.

It’s fundamental to how S3 works and its ability to scale, so it is a defining feature of S3.

If you think wider, a bucket itself is just a prefix.

Re: S3 is files, but not a filesystem

#204

Post by staunch »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

> Believe the hype.

I'd rather believe the test results.

Is there a neutral third-party that has validated S3's durability/integrity/consistency? Something as rigorous as Jepsen?

It'd be really neat if someone compared all the S3 compatible cloud storage systems in a really rigorous way. I'm sure we'd discover that there are huge scary problems. Or maybe someone already has?

Re: S3 is files, but not a filesystem

#205

Post by ahepp »

solumunus wrote: post #6
leetrout wrote: post #3

My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…

What exactly do you think a folder is? It’s just an abstraction for organising data.

Is it an abstraction for requesting the data you want, or an abstraction for storing the data in a retrievable manner?

Re: S3 is files, but not a filesystem

#206

Post by rsync »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

"AWS' availability zone isolation is better than the other cloud providers."

Not better than all of them.

A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver.

"S3 even operates at a scale where we could detect "bitrot""

That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale involved is tiny.

[1] he.net headquarters

Re: S3 is files, but not a filesystem

#207

Post by [deleted] »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

[deleted]

Re: S3 is files, but not a filesystem

#208

Post by Helmut10001 »

rsync wrote: post #206
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

Agree.

> S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc).

I would expect any cloud provider to be able to detect bitrot these days.

Re: S3 is files, but not a filesystem

#209

Post by ahepp »

Are filesystems the correct abstraction to build databases on? Isn’t a filesystem a database in a way? Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction?

To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstraction optimized for incompatible usages on top of that, you are going to have a difficult time.

Re: S3 is files, but not a filesystem

#210

Post by d-z-m »

> S3 is a cloud filesystem, not an object-whatever. [...]I think the idea that S3 is really "Amazon Cloud Filesystem" is a bit of a load bearing fiction.

Does anyone actually think this? I have never encountered anyone who has described S3 in these terms.

Post reply on HN