Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

351–360 of 456 posts

Re: S3 is files, but not a filesystem

#351

Post by chubot »

est31 wrote: post #319
chubot wrote: post #239

Earlier quoted context omitted.

Thanks for the link. But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used: https://www.postgresql.org/docs/current/non-durability.html Litestream documents the their data loss window, it seems like Neon should too: https://litestream.io/tips/ By default, Litestream will replicate new changes to an S3 replica every second. Dur…

> I searched the docs for "durability" and got zero results. The link I gave above explains it, right the sentence with "durability": > Safekeepers are responsible for durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to cloud storage. > Safekeepers can be thought of as an ultra…

OK thanks, I glossed over that part. But really what I look for is (1) explicit claims about durability, and (2) a third party (e.g. Aphyr) actually tested the claims.

If there's no claim, then it's impossible to test :)

In particular, there are no numbers in the description you quoted.

Litestream gives a relatively weak claim, but it could be tested, which actually gives me more confidence in it.

If you look at what aphyr writes, a lot of it is claims from vendors that turned out to be false - https://aphyr.com/tags/jepsen

Re: S3 is files, but not a filesystem

#352

Post by wetmore »

luhn wrote: post #305
returningfory2 wrote: post #220

Earlier quoted context omitted.

Are you performing list calls sequentially? If you have O(100k) directories and are doing O(100k) requests sequentially, 15 minutes works out at O(10ms) per request which doesn’t seem that bad? (assuming my math is correct…)

At risk of being pedantic, you seem to be using big O to mean “approximately” or “in the order of”, but that’s not what it means at all. Big O is an expression of the growth rate of a function. Any constant value has a growth rate of 0, so O(100k) isn’t meaningful: It’s exactly the same as O(1).

You're right technically, it's an abuse of notation that isn't uncommon. My physics profs would do it in college.

Re: S3 is files, but not a filesystem

#353

Post by yencabulator »

akerl_ wrote: post #350
yencabulator wrote: post #283

Earlier quoted context omitted.

You can't create new things on a read-only filesystem, you can in S3; not a good analogy.

I wasn’t making an analogy. I was asking how read-only filesystem works given the parent commenters description of what makes something a filesystem.

It's a filesystem where many operations return an error (historically, EROFS). There are many things you can't do with one. Is that interesting somehow?

I don't agree with defining a filesystem as something that has to be backed by a block device, but the shape of a filesystem API is historically very different from the shape of the S3 API.

Re: S3 is files, but not a filesystem

#354

Post by pl4nty »

overstay8930 wrote: post #219
cynicalsecurity wrote: post #110

Backblaze B2 is worth mentioning while we are speaking of S3. I'm absolutely in love with their prices (3 times lower than of S3). (I'm not their representative).

We liked B2 but not enough to pay for IPv4 addresses, insane they advertise as a multi-cloud solution but basically kill any chance at adoption when NAT gateways and IPv4 charges are everywhere. We would literally save money paying B2 bandwidth fees (high read low write) but not when being pushed through a NAT64 gateway, or paying an hourly charge just to be able to access B2.

they've started internal v6 rollout with external coming afterwards. no timelines though, and I've waited for years

https://old.reddit.com/r/backblaze/comments/1av4r3g/b2_ipv6_...

Re: S3 is files, but not a filesystem

#355

Post by tomcam »

bigiain wrote: post #348
cyberax wrote: post #335

Earlier quoted context omitted.

At least 3.

At least 3, in at least 3 seperate datacenters. According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3. I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if som…

> if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive.

Speak for yourself. Many of us love our audit logs and show them to strangers whatever we can.

Re: S3 is files, but not a filesystem

#356

Post by SirOibaf »

At Hopsworks we built HopsFS-S3 to improve things like listing (becomes a partition pruned scan of a in-memory DB), atomic renames and added a block/object caching layer using NVMe drives.

You can read the research paper here if you are curious: https://www.hopsworks.ai/research-papers/hopsfs-s3-extending...

Re: S3 is files, but not a filesystem

#357

Post by belter »

ianburrell wrote: post #251
belter wrote: post #211

Earlier quoted context omitted.

"Building and Maintaining an Amazon S3 Metadata Index without Servers" - https://aws.amazon.com/pt/blogs/big-data/building-and-mainta... Here is the architecture of Amazon Drive and the storage of metadata. "AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546

That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects. But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.

The problem was not the eventual consistency model, was the speed of the object list.

"...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."

Re: S3 is files, but not a filesystem

#358

Post by orf »

fijiaarone wrote: post #299
cuno wrote: post #162

Earlier quoted context omitted.

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

If you think s3 is fast, you should try FTP. It’s at least a hundred times faster. And combined with rsync, dozens of times more reliable.

Neither of those are true though? Not sure if this is sarcastic or not, if so make it more clear in the future

Re: S3 is files, but not a filesystem

#359

Post by orf »

belter wrote: post #357
ianburrell wrote: post #251

Earlier quoted context omitted.

That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects. But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.

The problem was not the eventual consistency model, was the speed of the object list. "...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."

It was both actually, but more for the listing issue. Netflix built a lot of tooling around this.

But yeah: things like filtering on tags or created at dates requires another approach.

Re: S3 is files, but not a filesystem

#360

Post by orf »

fijiaarone wrote: post #304
belter wrote: post #156

Earlier quoted context omitted.

No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

So in reality S3 takes about 2 seconds to retrieve a single file, under ideal conditions. 1 second round trip for the request to DynamoDB to get the object key of the file and 1 second round trip to S3 to get the file contents (assuming no CPU cost on the search because you’re getting the key by ID from the DynamoDB in a flat single table store. And that the file has no network IO because it is a trivial number of by…

2 seconds is a nuts response time, but I guess it depends entirely on your file size. TTFB is usually 50ms.
Post reply on HN