Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

361–370 of 456 posts

Re: S3 is files, but not a filesystem

#361

Post by orf »

yencabulator wrote: post #281
cuno wrote: post #168

Earlier quoted context omitted.

Actually we've found it's often much worse than that. Code written against AWS S3 using the AWS SDK often doesn't work on a great many "S3-compatible" vendors (including on-prem versions). Although there's documentation on S3, it's vague in many ways, and the AWS SDKs rely on actual AWS behaviour. We've had to deal with a lot of commercial and cloud vendors that subtly break things. This includes giant public cloud c…

Back in 2011 when I was working on making Ceph's RadosGW more S3-compatible, it was pretty common that AWS S3 behavior differed from their documentation too. I wrote a test suite to run against AWS and Ceph, just to figure out the differences. That lives on at https://github.com/ceph/s3-tests

What differences in behaviour from the AWS docs did you find, out of interest?

Re: S3 is files, but not a filesystem

#362

Post by dmvdoug »

tomcam wrote: post #355
bigiain wrote: post #348

Earlier quoted context omitted.

At least 3, in at least 3 seperate datacenters. According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3. I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if som…

> if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive. Speak for yourself. Many of us love our audit logs and show them to strangers whatever we can.

I’m picturing you having a slideshow of audit logs that you make guests to your home sit down and watch with you, like the vacation pictures slideshow of old.

Re: S3 is files, but not a filesystem

#363

Post by yencabulator »

orf wrote: post #361
yencabulator wrote: post #281

Earlier quoted context omitted.

Back in 2011 when I was working on making Ceph's RadosGW more S3-compatible, it was pretty common that AWS S3 behavior differed from their documentation too. I wrote a test suite to run against AWS and Ceph, just to figure out the differences. That lives on at https://github.com/ceph/s3-tests

What differences in behaviour from the AWS docs did you find, out of interest?

What I can dig up today is that back in 2011, they documented that bucket names cannot look like IPv4 addresses and the character set was a-z0-9.-, but they failed to prevent 192.168.5.123 or _foo.

I recall there were more edge cases around HTTP headers, but they don't seem to have been recorded as test cases -- it's been too long for me to remember details, I may have simply ran out of time / real world interop got good enough to prioritize something else.

2011 state, search for fails_on_aws: https://github.com/tv42/s3-tests/blob/master/s3tests/functio...

Current state, I can't speak to the exact semantics of the annotations today, they could simply be annotating non-AWS features: https://github.com/ceph/s3-tests/blob/master/s3tests/functio...

Re: S3 is files, but not a filesystem

#364

Post by returningfory2 »

luhn wrote: post #305
returningfory2 wrote: post #220

Earlier quoted context omitted.

Are you performing list calls sequentially? If you have O(100k) directories and are doing O(100k) requests sequentially, 15 minutes works out at O(10ms) per request which doesn’t seem that bad? (assuming my math is correct…)

At risk of being pedantic, you seem to be using big O to mean “approximately” or “in the order of”, but that’s not what it means at all. Big O is an expression of the growth rate of a function. Any constant value has a growth rate of 0, so O(100k) isn’t meaningful: It’s exactly the same as O(1).

Fair point, I guess the notation ~100k, ~10ms would be better.

Re: S3 is files, but not a filesystem

#365

Post by throwaway2037 »

kevincox wrote: post #349
fsociety wrote: post #292

Earlier quoted context omitted.

I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.

I think also Google as a whole has pretty good diversity. But Cloud customers demanded regions in big population centers and smaller countries where Google traditionally avoided due to cost reasons. This lead to less redundant sites that were often owned and/or operated by third parties. So in the US and Europe you can probably trust GCP zones quite literally. But other regions (I have heard lots of rumours in the AP…

> big population centers and smaller countries

Can we stop this dance on HN? Can you just name them, please?

Re: S3 is files, but not a filesystem

#366

Post by paulddraper »

donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

> Recursively listing these files

There's no "recursive" nature to S3 buckets. "Listing a directory" is simply listing keys by a prefix.

So list by the upper-most prefix that you want. If you have 1,000,000 files, it will take 1,000 API calls to list everything.

If each call takes 1s (I have no idea what your latency to the S3 bucket region is), then it will indeed take 15 min.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

Re: S3 is files, but not a filesystem

#367

Post by paulddraper »

somedudetbh wrote: post #241

> Amazon S3 is the original cloud technology: it came out in 2006. "Objects" were popular at the time and S3 was labelled an "object store", but everyone really knows that S3 is for files. S3 Alternative theory: everyone who worked on this knew that it was not a filesystem and "object store" is a description intended to describe everything else pointed out in this post. "Objects were really popular" is about objects…

Yeah, I'm really worried the author is confusing OOP with an object store.

To quote GCP:

> Object storage is a data storage architecture for storing unstructured data, which sections data into units—objects—and stores them in a structurally flat data environment

> https://cloud.google.com/learn/what-is-object-storage

That is (1) unstructured (2) flat organization (3) whole-item operations (read, write)

Re: S3 is files, but not a filesystem

#368

Post by paulddraper »

thinkharderdev wrote: post #245
alphazard wrote: post #180

S3 is not even files, and definitely not a filesystem. The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I ne…

> S3 is not even files, and definitely not a filesystem. I agree. To me the correct analog for S3 is a block storage device (a very weird one where blocks can be any size and can have a key associated with them) and not a filesystem. A filesystem is an abstraction that sits on top of a block storage device and so an "S3 filesystem" would have to be an abstraction that sits on top of S3 as the underlying block storage…

> a very weird one where blocks can be any size and can have a key associated with them

That is a very weird one

Re: S3 is files, but not a filesystem

#369

Post by paulddraper »

akerl_ wrote: post #183
alphazard wrote: post #180

S3 is not even files, and definitely not a filesystem. The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I ne…

How do read-only filesystems align with your definition?

All of the read stuff still applies (list, open, read, seek).

Re: S3 is files, but not a filesystem

#370

Post by paulddraper »

ein0p wrote: post #324

A bit off topic but also related: I use Minio as a local "S3" to store datasets and model checkpoints for my garage compute. Minio, however, has a bunch of features that I simply don't need. I just want to be able copy to/from, list prefixes, and delete every now and then. I could use nfs I suppose, but that'd be a bit inconvenient since I also use Minio to store build deps (which Bazel then downloads), and I'd like…

FYI, Minio used to have a "File System" mode that did exactly this.

But they deprecated it.

(You can still use it, but it's not getting updates.)

Post reply on HN