Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

281–290 of 456 posts

Re: S3 is files, but not a filesystem

#281

Post by yencabulator »

cuno wrote: post #168
bradleyjg wrote: post #130

Earlier quoted context omitted.

Code written against s3 is not portable either. It doesn’t support azure or gcp, much less some random proprietary cloud.

Actually we've found it's often much worse than that. Code written against AWS S3 using the AWS SDK often doesn't work on a great many "S3-compatible" vendors (including on-prem versions). Although there's documentation on S3, it's vague in many ways, and the AWS SDKs rely on actual AWS behaviour. We've had to deal with a lot of commercial and cloud vendors that subtly break things. This includes giant public cloud c…

Back in 2011 when I was working on making Ceph's RadosGW more S3-compatible, it was pretty common that AWS S3 behavior differed from their documentation too. I wrote a test suite to run against AWS and Ceph, just to figure out the differences. That lives on at https://github.com/ceph/s3-tests

Re: S3 is files, but not a filesystem

#282

Post by pclmulqdq »

senderista wrote: post #262
Helmut10001 wrote: post #208

Earlier quoted context omitted.

Agree. > S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc). I would expect any cloud provider to be able to detect bitrot these days.

I think the point the OP was trying to make is that they regularly detected bitrot due to their scale, not that they were merely capable of doing so.

Everyone with significant scale and decent software regularly detects bitrot.

Re: S3 is files, but not a filesystem

#283

Post by yencabulator »

akerl_ wrote: post #183
alphazard wrote: post #180

S3 is not even files, and definitely not a filesystem. The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I ne…

How do read-only filesystems align with your definition?

You can't create new things on a read-only filesystem, you can in S3; not a good analogy.

Re: S3 is files, but not a filesystem

#284

Post by alexchamberlain »

breckognize wrote: post #216
rsync wrote: post #206

Earlier quoted context omitted.

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver. The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough tha…

> They're far enough apart for good isolation, which provides durability and availability

It can't possibly be enough for critical data though, right? I'm guessing a fire in 1 is unlikely to spread to another, but could it affect the availability of another? What about a deliberate attack on the DCs or the utilities supplying the DCs?

Re: S3 is files, but not a filesystem

#285

Post by jandrewrogers »

ahepp wrote: post #209

Are filesystems the correct abstraction to build databases on? Isn’t a filesystem a database in a way? Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction? To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstr…

The traditional POSIX filesystem is the wrong abstraction for a database, but not filesystems per se. All databases that care about performance and scalability implement their own filesystems, either directly against raw block devices or as an overlay on top of a POSIX filesystem that bypasses some of its limitations. The performance and scalability gains by doing so are not small.

The issue with POSIX filesystems is that they are required to make a set of tradeoffs to support features a database engine doesn't need, to the significant detriment of scalability and performance in areas that databases care about a lot. For example, one such database filesystem I've used occasionally over the years, while a bit dated at this point, is designed such that you can have tens of millions of files in a single directory where you are creating and destroying tens of thousands of files every second, on upwards of a petabyte of storage. Very far from being POSIX compatible but you don't get anything like that type of scalability on POSIX.

Object storage is far from ideal as database storage. The biggest issue, though, is the terrible storage bandwidth available in the cloud. It is a small fraction of what is available in a normal server and modern database engines are capable of fully exploiting a large JBOD of NVMe.

Re: S3 is files, but not a filesystem

#286

Post by electroly »

donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

The way that you said "recursively" and spent a lot of time describing "directories" and "levels" worries me. The fastest way to list objects in S3 wouldn't involve recursion at all; you just list all objects under a prefix. If you're using the path delimiter to pretend that S3 keys are a folder structure (they're not) and go "folder by folder", it's going to be way slower. When calling ListObjectsV2, make sure you are NOT passing "delimiter". The "directories" and "levels" have no impact on performance when you're not using the delimiter functionality. Split the one list operation into multiple parallel lists on separate prefixes to attain any total time goal you'd like.

Re: S3 is files, but not a filesystem

#287

Post by pclmulqdq »

breckognize wrote: post #266
pclmulqdq wrote: post #253

Earlier quoted context omitted.

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us. As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and…

That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters).

I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.

Re: S3 is files, but not a filesystem

#288

Post by yencabulator »

BirAdam wrote: post #232

Underneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.

Ceph's BlueStore has talked direct to block devices, no filesystem in between, since 2017.

https://ceph.com/community/new-luminous-bluestore/

[Disclaimer: ex-Ceph employee, from before BlueStore]

Re: S3 is files, but not a filesystem

#289

Post by jandrewrogers »

BirAdam wrote: post #232

Underneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.

I seriously doubt this is correct. It is common for database engines to install directly on raw block devices, bypassing the Linux kernel and effectively becoming the filesystem for those storage devices. Why would S3 work any differently? There are no advantages to building on top of a filesystem and many disadvantages for this kind of thing.

It would be a poor engineering choice to build something like S3 on top of some other filesystem. There are often ways to do it by using an overlay that converts a filesystem into a pseudo block device, but that is usually considered a compatibility shim used for environments that lacking dedicated storage, at the cost of robustness and performance.

Re: S3 is files, but not a filesystem

#290

Post by kiwijamo »

ForHackernews wrote: post #96
vbezhenar wrote: post #71

Earlier quoted context omitted.

S3 usually is the cheapest storage, not only for Amazon, but for other clouds. I don’t understand why.

This is not true in my experience https://www.backblaze.com/cloud-storage/pricing

That Backblaze page (not surprisingly) compares their prices to a fairly expensive S3 pricing tier and makes other assumptions in Blackblaze's favour. For some use cases B2 is more expensive e.g. one copy of my backups goes to AWS Deep Glacier which is really cheap.
Post reply on HN