Random note: Has anyone noticed how fast the author's webpage is? I know it's static, but I mean it's fast even for the DNS lookup. I would love to know what they have on.
Full stack Cloudflare is really fast
S3 is files, but not a filesystem
221–230 of 456 posts
Re: S3 is files, but not a filesystem
#222Earlier quoted context omitted.
EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.
if you turn all the EFS performance knobs up (at a high cost), it's quite fast.
Faster, sure. But I wouldn’t got so far as to say it is fast
Re: S3 is files, but not a filesystem
#223> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…
> customers would beat us up over pricing compared to GCP blob storage, but the comparison was unfair because Google would store your data in the same building I don’t think this is true. Per the Google Cloud Storage docs, data is replicated across multiple zones, and each zone maps to a different cluster. https://cloud.google.com/compute/docs/regions-zones/zone-vir...
Google puts multiple clusters in a single building.
Re: S3 is files, but not a filesystem
#224> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…
I think it’s far more mundane a reason. You can list 10k objects per request and getting the next 10k requires the result of the previous request, so it’s all serial. That means to list 1M files, you’re looking at 100 back to back requests. Assuming a ping time of 50ms, that’s easily 5s of just going back and forth, not including the cost of doing the listing itself on a flat iteration. The cost of a 10k item list is about the cost of a write which is kinda slow. Additionally, I suspect each listing is a strongly consistent snapshot which adds to the cost of the operation (it can be hard to provide an inconsistent view).
I don’t think btrees would help unless you’re doing directory traversals, but even then I suspect that’s not that beneficial as your bottleneck is going to be the network operations and exposed operations. Ultimately, file listing isn’t that critical a use case and typically most use cases are accomplished through things like object lifecycles where you tell S3 what you want done and it does it efficiently at the FS layer for you.
Re: S3 is files, but not a filesystem
#225Backblaze B2 is worth mentioning while we are speaking of S3. I'm absolutely in love with their prices (3 times lower than of S3). (I'm not their representative).
We liked B2 but not enough to pay for IPv4 addresses, insane they advertise as a multi-cloud solution but basically kill any chance at adoption when NAT gateways and IPv4 charges are everywhere. We would literally save money paying B2 bandwidth fees (high read low write) but not when being pushed through a NAT64 gateway, or paying an hourly charge just to be able to access B2.
How could they launch a cloud service like this and not have IPv6 in 2015? What other basic things did they cheap out on?
Re: S3 is files, but not a filesystem
#226> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…
The key difference between lexicographically keyed flat hierarchies, and directory-nested filesystem hierarchies, becomes clear based on this example:
dir1/a/000000
dir1/a/...
dir1/a/999999
dir1/b
A flat string-indexed KV store that only supports lexicographic order, without special handling of delimters, needs to traverse 1 million dirents ("a/00000" throuh "a/999999") before arriving at "b".
Thus, simple flat hierarchies are much slower at listing the contents of a single dir: O(all recursive children), vs. O(immediate children) on a "proper" filesystem.
Lexicographic strings cannot model multi-level tree structures with the same complexities; this may give it the reputation of "listing files is slow".
UNLESS you tell the listing algorithm what the delimter character is (e.g. `/`). Then a lexicographical prefix tree can efficiently skip over all subtrees at the next `/`.
Amazon S3 supports that, with the docs explicitly mentioning "skipping over and summarizing the (possibly millions of) keys nested at deeper levels" in the `CommonPrefixes` field: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
I have not tested whether Amazon's implemented actually saves the traversal (or whether it traverses and just returns less results), but I'd hope so.
Re: S3 is files, but not a filesystem
#227> Filesystem software, especially databases, can't be ported to Amazon S3 This seems mistaken. Porting databases that run on local disk to S3 seems like a good way to get a lashing from https://aphyr.com/ Can any databases do it correctly? If so, I doubt they work with the model of partial overwrites. They probably have to do something very custom, and either sacrifice a lot of tail latency, or their uptime is capped…
My employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/architecture-overview
Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside of S3 in a replicated on-disk cache.
Re: S3 is files, but not a filesystem
#228Earlier quoted context omitted.
> customers would beat us up over pricing compared to GCP blob storage, but the comparison was unfair because Google would store your data in the same building I don’t think this is true. Per the Google Cloud Storage docs, data is replicated across multiple zones, and each zone maps to a different cluster. https://cloud.google.com/compute/docs/regions-zones/zone-vir...
Google puts multiple clusters in a single building.
Seems you’re right. They say each zone is a separate failure domain but you kind of have to trust their word on that.
Re: S3 is files, but not a filesystem
#229The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…
I’ve wondered this also because it can be handy to have multiple ways of accessing the same file. For example to obfuscate database uuids if they are used in the key. In theory you could implement soft links in AWS by just storing a file with the path to the linked file. But it would be a lot of manual work.
Re: S3 is files, but not a filesystem
#230Earlier quoted context omitted.
Constant time irregardless of the number of objects in the bucket and irregardless of the initial starting position of your list request.
The technical implementation is indeed impressive that it operates more-or-less within constant time, but probably very few use cases actually fit that narrow window, so this technical strength is moot when it comes to actual usage. Since each request is dependent upon the position received in the last request, 1000 arbitrary keys on your 3rd or 1000th attempt doesn't really help unless you found your needle in the h…
You’re assuming you’re paginating through all objects from start to finish.
A request to list objects under “foo/“ is a request to list all objects starting with “foo/“, which is constant time irregardless of the number of keys before. Same applies for “foo/bar-“, or any other list request for any given prefix. There are no directories on s3.