Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

311–320 of 456 posts

Re: S3 is files, but not a filesystem

#311

Post by anonymous-panda »

tsimionescu wrote: post #264
anonymous-panda wrote: post #224

Earlier quoted context omitted.

I think it’s far more mundane a reason. You can list 10k objects per request and getting the next 10k requires the result of the previous request, so it’s all serial. That means to list 1M files, you’re looking at 100 back to back requests. Assuming a ping time of 50ms, that’s easily 5s of just going back and forth, not including the cost of doing the listing itself on a flat iteration. The cost of a 10k item list is…

That's 5s of a 15m duration. I don't think it matters in the least.

Depends how you’re iterating. If your iterating by hierarchy level, then you could easily see this being several orders of magnitude more requests.

Re: S3 is files, but not a filesystem

#312

Post by anonymous-panda »

catlifeonmars wrote: post #271
anonymous-panda wrote: post #259

Earlier quoted context omitted.

Directories make up a hierarchical filesystem, but it’s not a necessary condition. A filesystem at its core is just a way of organizing files. If you’re storing and organizing files in s3 then it’s a filesystem for you. Saying it’s “fundamentally a key value store” like it’s something different is confusing because a filesystem is just a key value store of path to contents of file. Indeed there’s every reason to beli…

In context of the comment I was addressing, it’s clear that filesystem means more than just a key value store. I’d argue that this is generally true in common vernacular.

This is a technical website discussing the nuances of filesystems. Common vernacular is how you choose to define it but even the Wikipedia definition says that directories and hierarchy are just one property of some filesystems. That they became the dominant model on local machines doesn’t take away from the more general definition that can describe distributed filesystems.

Re: S3 is files, but not a filesystem

#313

Post by jameshart »

electroly wrote: post #309
jameshart wrote: post #302

Earlier quoted context omitted.

A fun corollary of this issue: Deleting an S3 bucket is nontrivial! You can't delete a bucket with objects in it. And you can't just tell S3 to delete all the objects. You need to send individual API requests to S3 to delete each object. Which means sending requests to S3 to list out the objects, 1000 at a time. Which takes time. And those list calls cost money to execute. This is a good summary of the situation: htt…

No, don't do that. Set up a lifecycle rule that expires all of the objects and wait 24 hours. You won't pay for API calls and even the cost of storing the objects themselves is waived once they are marked for expiration. The article has a mistake about this too: expirations do NOT count as lifecycle transitions and you don't get charged as such. You will, of course, get charged if you prematurely delete objects that…

Still counts as nontrivial.

Re: S3 is files, but not a filesystem

#314

Post by electroly »

jameshart wrote: post #313
electroly wrote: post #309

Earlier quoted context omitted.

No, don't do that. Set up a lifecycle rule that expires all of the objects and wait 24 hours. You won't pay for API calls and even the cost of storing the objects themselves is waived once they are marked for expiration. The article has a mistake about this too: expirations do NOT count as lifecycle transitions and you don't get charged as such. You will, of course, get charged if you prematurely delete objects that…

Still counts as nontrivial.

This is really easy; much easier than trying to delete them by hand. AWS does all the work for you. It takes longer to log into the AWS Management Console than it does to set up this lifecycle rule.

Re: S3 is files, but not a filesystem

#315

Post by duckman1 »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

[dead]

Re: S3 is files, but not a filesystem

#316

Post by prima-facie »

chrisblackwell wrote: post #177

Random note: Has anyone noticed how fast the author's webpage is? I know it's static, but I mean it's fast even for the DNS lookup. I would love to know what they have on.

Could be using Cloudflare pages hosted on a R2 bucket: https://pages.cloudflare.com/

Re: S3 is files, but not a filesystem

#317

Post by blakesley »

electroly wrote: post #286
donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

The way that you said "recursively" and spent a lot of time describing "directories" and "levels" worries me. The fastest way to list objects in S3 wouldn't involve recursion at all; you just list all objects under a prefix. If you're using the path delimiter to pretend that S3 keys are a folder structure (they're not) and go "folder by folder", it's going to be way slower. When calling ListObjectsV2, make sure you a…

All these comments saying merely "S3 has no concept of directories" without an explanation (or at least a link to an explanation) are pretty unhelpful, IMO. I dismissed your comment, but then I came upon this later one explaining why: https://news.ycombinator.com/item?id=39660445

After reading that, I now understand your comment.

Re: S3 is files, but not a filesystem

#318

Post by woodada »

mannyv wrote: post #279
rsync wrote: post #206

Earlier quoted context omitted.

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

How does the latest ZFS bug impact your bitrot statement? I mean, technically it’s not bitrot if zeros were accidentally written out instead of data.

Probably none because they didn't update to the exact version that had the bug

Re: S3 is files, but not a filesystem

#319

Post by est31 »

chubot wrote: post #239
est31 wrote: post #227

Earlier quoted context omitted.

My employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/architecture-overview Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside o…

Thanks for the link. But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used: https://www.postgresql.org/docs/current/non-durability.html Litestream documents the their data loss window, it seems like Neon should too: https://litestream.io/tips/ By default, Litestream will replicate new changes to an S3 replica every second. Dur…

> I searched the docs for "durability" and got zero results.

The link I gave above explains it, right the sentence with "durability":

> Safekeepers are responsible for durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to cloud storage.

> Safekeepers can be thought of as an ultra reliable write buffer that holds the latest data until it is processed and uploaded to cloud storage. Safekeepers implement the Paxos protocol for reliability.

Re: S3 is files, but not a filesystem

#320

Post by breckognize »

pclmulqdq wrote: post #287
breckognize wrote: post #266

Earlier quoted context omitted.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and…

That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters). I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.

Do you mean Google included "acts of God" when computing 9's? That's definitely not right.

11 9's of durability means mean time to data loss of 100 billion years. Nothing on earth is 11 9's durable in the face of natural (or man-made) disasters. The earth is only 4.5 billion years old.

Post reply on HN