Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

251–260 of 456 posts

Re: S3 is files, but not a filesystem

#251

Post by ianburrell »

belter wrote: post #211
macintux wrote: post #190

Earlier quoted context omitted.

I don't know that you can characterize that as a "standard practice". Maybe it's widespread, but I've not encountered it.

"Building and Maintaining an Amazon S3 Metadata Index without Servers" - https://aws.amazon.com/pt/blogs/big-data/building-and-mainta... Here is the architecture of Amazon Drive and the storage of metadata. "AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546

That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects.

But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.

Re: S3 is files, but not a filesystem

#252

Post by navaati »

singron wrote: post #223
medler wrote: post #212

Earlier quoted context omitted.

> customers would beat us up over pricing compared to GCP blob storage, but the comparison was unfair because Google would store your data in the same building I don’t think this is true. Per the Google Cloud Storage docs, data is replicated across multiple zones, and each zone maps to a different cluster. https://cloud.google.com/compute/docs/regions-zones/zone-vir...

Google puts multiple clusters in a single building.

Flashback to that Clichy datacenter fire near Paris...

Re: S3 is files, but not a filesystem

#253

Post by pclmulqdq »

treflop wrote: post #231
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough. The things that you mentioned are standard storage company activities. Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it. Keepin…

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us.

As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

Re: S3 is files, but not a filesystem

#254

Post by Spivak »

catlifeonmars wrote: post #235
donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

S3 is fundamentally a key value store. The fact that you can view objects in “directories” is nothing more than a prefix filter. It is not a file system and has no concept of directories.

If I wanted to use S3 as a filesystem in the manner people are describing I would probably start looking at storing filesystem metadata in a sidecar database so you can get directory listings, permissions bits, xattrs and only have to round-trip to S3 when you need the content.

Re: S3 is files, but not a filesystem

#255

Post by logifail »

emodendroket wrote: post #246
logifail wrote: post #240

Earlier quoted context omitted.

> The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle What may be less of a sweet spot is AWS' pricing.

Sending the data to /dev/null is the cheapest option if that’s all you care about.

Seems the snark detector just went off :)

Back on topic, I'd hope all of us would expect value for money for any and all services we recommend or purchase. Search for "site:news.ycombinator.com Away From AWS" to find dozens of discussions on how to save money by leaving AWS.

EDIT: just one article of the many I've read recently:

"What I’ve always found surprising about egress is just how expensive it is. On AWS, downloading a file from S3 to your computer once costs 4 times more than storing it for an entire month"

https://robaboukhalil.medium.com/youre-paying-too-much-for-e...

Re: S3 is files, but not a filesystem

#256

Post by nh2 »

nh2 wrote: post #226
orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

The key difference between lexicographically keyed flat hierarchies, and directory-nested filesystem hierarchies, becomes clear based on this example: dir1/a/000000 dir1/a/... dir1/a/999999 dir1/b On a proper hierarchical file file system with directories as tree interior nodes, `ls dir1/` needs to traverse and return only 2 entries ("a" and "b"). A flat string-indexed KV store that only supports lexicographic order,…

For completeness: The orignal post says:

    S3 has no rename or move operation.
    Renaming is CopyObject and then DeleteObject.
    CopyObject takes linear time to the size of the file(s).
    This comes up fairly often when someone has written a lot of files
    to the wrong place - moving the files back is very slow.
This is right:

In a normal file system, renaming a directory is fast O(1), in S3 it's slow O(all recursive children).

And Amazon S3 has not added a delimiter-based function to reduce its complexity, even though that would be easily possible in a lexicographic prefix tree (re-rooting the subtree).

So here the original post has indeed found a case where S3 is much slower than a normal file system.

Re: S3 is files, but not a filesystem

#257

Post by throwaway892238 »

svat wrote: post #198
throwaway892238 wrote: post #45

> The "simple" in S3 is a misnomer. S3 is not actually simple. It's deep. Simple doesn't mean "not deep". It means having the fewest parts needed in order to accomplish your requirements. If you require a distributed, centralized, replicated, high-availability, high-durability, high-bandwidth, low-latency, strongly-consistent, synchronous, scalable object store with HTTP REST API, you can't get much simpler than S3.…

> It means having the fewest parts needed in order to accomplish your requirements. That is exactly what "deep" means, in the terminology of this post (from Ousterhout's book A Philosophy of Software Design ). Simple means "not complex" (see also Rich Hickey's talk Simple Made Easy: https://www.infoq.com/presentations/Simple-Made-Easy/ ), while "deep" means providing/having a lot of internally-complex functionality v…

Hickey's definition of simple is wrong. It's not the opposite of complex at all. They are not opposites, nor mutually exclusive.

  - Easy is when something does not require much effort.
  - Simple means the least complex it can be and still work.
  - Complex means there are lots of components.
These are all quite different concepts:

  - Easy is a concept that distinguishes the amount of work needed to use a solution
  - Simple is a concept that distinguishes whether or not there is an excess number of interacting properties in a system
  - Complex is a concept describing the quality of having a number of interacting properties in a system
Hickey's talk is useful in terms of thinking about software, but it also contains many over-generalizations which are incorrect and lead to incorrect thinking about things that aren't software. (Even some of his declarations about software are wrong)

"Deep", in the context of software complexity, probably only makes sense in terms of describing the number of layers involved in a piece of technology. You could make something have many layers, and it could still be simple, or be complex, or easy.

Re: S3 is files, but not a filesystem

#258

Post by perryizgr8 »

donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

It's not a good model to think of S3 has having directories in a bucket. It's all objects. The web interface has a visual way of representing prefixes separated by slashes. But that's just a nice way to present the objects. Each object has a key, and that key can contain slashes, and you can think of each segment to be a directory for your ease of mind.

But that illusion breaks when you try to do operations you usually do with/on directories.

Re: S3 is files, but not a filesystem

#259

Post by anonymous-panda »

catlifeonmars wrote: post #235
donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

S3 is fundamentally a key value store. The fact that you can view objects in “directories” is nothing more than a prefix filter. It is not a file system and has no concept of directories.

Directories make up a hierarchical filesystem, but it’s not a necessary condition. A filesystem at its core is just a way of organizing files. If you’re storing and organizing files in s3 then it’s a filesystem for you. Saying it’s “fundamentally a key value store” like it’s something different is confusing because a filesystem is just a key value store of path to contents of file.

Indeed there’s every reason to believe that a modern file system would perform significantly faster if the hierarchy was implemented as a prefix filter than actually maintaining the hierarchical data structures (at least for most operations). You can guess that this might be the case that file creation is extremely slow on modern file systems (on the order of hundreds or maybe thousands per second on a modern NVME disk that can otherwise do millions of IOPs and listing the contents of an extremely large directory is exceedingly slow)

Re: S3 is files, but not a filesystem

#260

Post by senderista »

breckognize wrote: post #216
rsync wrote: post #206

Earlier quoted context omitted.

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver. The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough tha…

> the network round trip time is negligible compared to the disk seek

Only for spinning rust, right?

Post reply on HN