Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

411–420 of 456 posts

Re: S3 is files, but not a filesystem

#411

Post by jmull »

anonymous-panda wrote: post #312
catlifeonmars wrote: post #271

Earlier quoted context omitted.

In context of the comment I was addressing, it’s clear that filesystem means more than just a key value store. I’d argue that this is generally true in common vernacular.

This is a technical website discussing the nuances of filesystems. Common vernacular is how you choose to define it but even the Wikipedia definition says that directories and hierarchy are just one property of some filesystems. That they became the dominant model on local machines doesn’t take away from the more general definition that can describe distributed filesystems.

I'm kind of chuckling at this thread because you're working so hard to not understand.

I think the previous poster could/should have said, "It is not a hierarchical file system and has no concept of directories." where I added the word "hierarchical".

But it's also pretty obvious that was the point.

Re: S3 is files, but not a filesystem

#412

Post by geertj »

hnlmorg wrote: post #181
cuno wrote: post #172

Earlier quoted context omitted.

For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/ and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...

EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.

> EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.

Would you care to elaborate on your experience or use case a bit more? We've made a lot of improvements over the last few years (and are actively working on more), and we have many happy customers. I'd be happy to give a perspective of how well your use case would work with EFS.

Source: PMT turned engineer on EFS, with the team for over 6 years

Re: S3 is files, but not a filesystem

#413

Post by wubrr »

pclmulqdq wrote: post #253
treflop wrote: post #231

Earlier quoted context omitted.

What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough. The things that you mentioned are standard storage company activities. Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it. Keepin…

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us. As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

> As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

How can you tell that if it's not measurable?

As far as I can tell the '11/14 9s' durability numbers are more or less completely made up. That's why AWS doesn't offer any actual durability SLA for S3, only a 99.9% availability SLA[0].

[0] https://aws.amazon.com/s3/sla/

Re: S3 is files, but not a filesystem

#414

Post by wubrr »

breckognize wrote: post #266
pclmulqdq wrote: post #253

Earlier quoted context omitted.

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us. As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and…

9's are useful when they're backed by an actual SLA - like GCP Cloud Storage and AWS S3 availability SLAs. Neither one commits to any durability SLAs whatsoever so I wouldn't put any stock into the 'eleventy nine nines' durability claims.

Re: S3 is files, but not a filesystem

#415

Post by tom910 »

bigiain wrote: post #348
cyberax wrote: post #335

Earlier quoted context omitted.

At least 3.

At least 3, in at least 3 seperate datacenters. According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3. I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if som…

I think lees. Why AWS need to store 3 times if they can use Reed-Solomon algorithms (or similar) and decrease this number to 2 or 1.5 and save a lot of storage space

Re: S3 is files, but not a filesystem

#416

Post by calpaterson »

svat wrote: post #202

It's nice to see Ousterhout's idea of module depth (the main idea from his A Philosophy of Software Design ) getting more mainstream — mentioned in this article with attribution only in "Other notes", which suggests the author found it natural enough not to require elaboration. Being obvious-in-hindsight like this is a sign of a good idea. :-) > The concept of deep vs shallow modules comes from John Ousterhout's exce…

I was tempted to define and cite the term more carefully before I used it but that leadened the article a lot right in the middle and so I cut it and just hoped.

It is a great concept and also a great book. I really enjoyed it but I've never found a convincing way to persuade people to read it. I read it on personal recommendation but that only works if it comes from someone you respect (as in my case).

Re: S3 is files, but not a filesystem

#417

Post by calpaterson »

ahepp wrote: post #209

Are filesystems the correct abstraction to build databases on? Isn’t a filesystem a database in a way? Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction? To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstr…

> Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction?

Oracle DB for a long time supported running on raw partitions which I think suggests that the answer is "not really". Snowflake (and I hear Clickhouse) can run on S3 which I think is more evidence against running on a filesystem. Not to mention the torrid time Postgres has had with fsync on linux.

Re: S3 is files, but not a filesystem

#418

Post by callalex »

pclmulqdq wrote: post #294
fsociety wrote: post #292

Earlier quoted context omitted.

I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.

Yeah, that's definitely true. Google sort of mapped an AWS concept onto its own cluster splits. However, there are enough regional-scale outages at all the major clouds that I don't personally place much stock in the idea of zones to begin with. The only way to get close to true 24/7 five-9's uptime with clouds is to be multi-region (and preferably multi-cloud).

I have experienced many outages that were contained to a specific availability zone in AWS, from power failures to flooding to cable cuts. You are correct that 5 9’s still requires multi-region though.

Re: S3 is files, but not a filesystem

#420

Post by calpaterson »

overstay8930 wrote: post #221
chrisblackwell wrote: post #177

Random note: Has anyone noticed how fast the author's webpage is? I know it's static, but I mean it's fast even for the DNS lookup. I would love to know what they have on.

Full stack Cloudflare is really fast

There is an nginx server running on debian stable somewhere in the dark heart of Germany. But I do have numerous tricks (too many, probably) to keep things quick.

But there are still ways to be quicker. For example, the header photo is smaller than the vector diagrams on the page, by about tenfold.

Post reply on HN