Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

231–240 of 456 posts

Re: S3 is files, but not a filesystem

#231

Post by treflop »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

What’s your experience like at other storage outfits?

I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough.

The things that you mentioned are standard storage company activities.

Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it.

Keeping track of hard drive failure rates by vendor is normal. Storage companies publicly publish their own reports. The tiny 6-person IT operation I was in had a spreadsheet. Hell, I toured a friend’s friend’s major data center last year and he managed to find time to talk hard drive vendors. Now you. I get it — y’all make spreadsheets.

There are a lot of smart people working on storage outside AWS and long before AWS existed.

Re: S3 is files, but not a filesystem

#232

Post by BirAdam »

Underneath the software, there’s still a filesystem with files.

If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.

Re: S3 is files, but not a filesystem

#233

Post by catlifeonmars »

supriyo-biswas wrote: post #196
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

Checksumming the data is not based out of paranoia but simply as a result of having to detect which blocks are unusable in order to run the Reed-Solomon algorithm. I'd also assume that a sufficient number of these corruption events are used as a signal to "heal" the system by migrating the individual data blocks onto different machines. Overall, I'd say the things that you mentioned are pretty typical of a storage sy…

The S3 checksum feature applies to the objects, so that’s entirely orthogonal to erasure codes. Unless you know something I don’t and SHA256 has commutative properties. You’d still need to compute the object hash independent of any blocks.

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...

Re: S3 is files, but not a filesystem

#234

Post by albert_e »

hn72774 wrote: post #173
albert_e wrote: post #58

Earlier quoted context omitted.

There is specifically block storage service (EBS) and falvirs of it like EBS multi-attach and EFS that can ne used if there is a need to port software/databases to the cloud with low level filesystem support. Why would we need to do it on object storage which addresses a different type of storage need. Nevertheless there are projects like EMRFS and S3 file system mount points that try to provide files stem interfaces…

S3 is better for large datasets. It's cheaper and handles large file sizes with ease. It has become a de-facto standard for distributed, data-intensive workloads like those common with spark. A key benefit is decoupling the data from the compute so that they can scale independently. EBS is tightly coupled to iops and you pay extra for that. (Source: a long time working in data engineering)

Yes and I also believe:

Experienced Spark / Data Engineering teams would not assume S3 is readily useable as a filesystem.

This [1] seems like a good guide on how to configure spark for working with Cloud object stores, while recognizing the limitations and pitfalls.

[1]: https://spark.apache.org/docs/latest/cloud-integration.html

---

Amazon EMR offers a managed way to run hadoop or spark clusters and it implements an "EMR FS" [2] system to interface with S3 as storage.

[2]: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-fs.h...

AWS Glue is another option which is "serverless" ETL. Source and Destination can be S3 data lakes read through a data catalog (hive or glue data catalog). During processing AWs Glue can optionally use S3 [3,4,5] for shuffle partition.

[3]: https://aws.amazon.com/blogs/big-data/introducing-amazon-s3-...

[4]: https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shu...

[5]: https://aws.amazon.com/blogs/big-data/introducing-the-cloud-...

Re: S3 is files, but not a filesystem

#235

Post by catlifeonmars »

donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

S3 is fundamentally a key value store. The fact that you can view objects in “directories” is nothing more than a prefix filter. It is not a file system and has no concept of directories.

Re: S3 is files, but not a filesystem

#236

Post by loeg »

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

Not a public cloud, but storage at Facebook is similar in terms of physical infrastructure, safety culture, and scale.

Re: S3 is files, but not a filesystem

#237

Post by cuno »

remram wrote: post #187

I am currently pondering this exact problem. I want to run a file-sharing web application (think: NextCloud) but I don't want to use expensive block storage or the dedicated server's disk space for the files, as some of them will be accessed infrequently. I am wondering if s3fs/rclone-mount is sufficient, or if I should use something like JuiceFS that adds random-access, renaming, etc on top of it. Are those really n…

It depends on if you want to expose filesystem semantics or metadata to applications using it. For example random access writes are done by ffmpeg, which is a workhorse of the media industry, but most things can't handle that or are too slow. We had to build our own solution cunoFS to make it work properly at high speeds.

Re: S3 is files, but not a filesystem

#238

Post by mechanicalpulse »

orf wrote: post #124
jacobsimon wrote: post #121

Earlier quoted context omitted.

What is it about S3 that enables this speed, and why can’t traditional Unix file systems do the same?

S3 doesn’t have directories, it could be thought of a flat + sorted list of keys. UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories. So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Isn't that a limitation imposed by the POSIX APIs, though, as a direct consequence of the interface's representation of hierarchical filesystems as trees? As you've illustrated, that necessitates walking the tree. Many tools, I suppose, walk the tree via a single thread, further serializing the process. In an admittedly haphazard test, I ran `find(1)` on ext4, xfs, and zfs filesystems and saw only one thread.

I imagine there's at least one POSIX-compatible file system out there that supports another, more performant method of dumping its internal metadata via some system call or another. But then we would no longer be comparing the S3 and POSIX APIs.

Re: S3 is files, but not a filesystem

#239

Post by chubot »

est31 wrote: post #227
chubot wrote: post #213

> Filesystem software, especially databases, can't be ported to Amazon S3 This seems mistaken. Porting databases that run on local disk to S3 seems like a good way to get a lashing from https://aphyr.com/ Can any databases do it correctly? If so, I doubt they work with the model of partial overwrites. They probably have to do something very custom, and either sacrifice a lot of tail latency, or their uptime is capped…

My employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/architecture-overview Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside o…

Thanks for the link.

But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used:

https://www.postgresql.org/docs/current/non-durability.html

Litestream documents the their data loss window, it seems like Neon should too:

https://litestream.io/tips/

By default, Litestream will replicate new changes to an S3 replica every second. During this time where data has not yet been replicated, a catastrophic crash on your server will result in the loss of data in that time window.

I also searched for "data loss" and got zero results -- this is important because Neon is almost certainly sacrificing durability for performance.

Re: S3 is files, but not a filesystem

#240

Post by logifail »

breckognize wrote: post #216
rsync wrote: post #206

Earlier quoted context omitted.

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver. The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough tha…

> The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle

What may be less of a sweet spot is AWS' pricing.

Post reply on HN