Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

401–410 of 456 posts

Re: S3 is files, but not a filesystem

#401

Post by EraYaN »

Filligree wrote: post #307
FooBarWidget wrote: post #303

Earlier quoted context omitted.

"Checksum-all-the-things is a basic feature of a lot of file systems" "A lot"? Does anything but ZFS and maybe btrfs do this? Ext4 anf XFS — two very common filesystems — still don't have data checksums.

Bcachefs, and LVM also has a way to do it. Unfortunately I’m not aware of any filesystem that does it while maintaining the full bandwidth of a modern NVMe. Not even with the extra reads factored in; on ZFS I get 800 MB/s max.

ZFS should absolutely be able to go faster even with lz4 compression I get writes above 5 GB/s on a older 32-core EPYC CPU. And that is with mostly random and already compressed data. And that write speed is a limitation of the RAIDz2 on top of not the fastest drives (6 PCIe 3.0 intel U.2 ones from 2 years ago).

Re: S3 is files, but not a filesystem

#402

Post by harshaw »

hnlmorg wrote: post #181
cuno wrote: post #172

Earlier quoted context omitted.

For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/ and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...

EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.

Have you tried it recently? Because we've made it a lot faster over the years.

Re: S3 is files, but not a filesystem

#403

Post by jacobsimon »

orf wrote: post #124
jacobsimon wrote: post #121

Earlier quoted context omitted.

What is it about S3 that enables this speed, and why can’t traditional Unix file systems do the same?

S3 doesn’t have directories, it could be thought of a flat + sorted list of keys. UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories. So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Interesting - isn't this just a matter of indexing/caching the file names, though? Surely S3 must store the files somewhere and index them. There's a Unix command called `locate` that does the same thing by maintaining a local database of keys and lets you search with prefixes.[1]

Anyway, I guess this is beyond the point of the original commenter above. I would disagree that listing files efficiently is the most useful part of S3. The main value prop is the fact that you can easily upload and download files from a distributed store. Most use cases involve uploading and downloading known files, not efficiently listing millions of files.

[1] https://jvns.ca/blog/2015/03/05/how-the-locate-command-works...

Re: S3 is files, but not a filesystem

#404

Post by harshaw »

aeyes wrote: post #127
orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

And if for some reason you need a complete listing along with object sizes and other attributes you can get one every 24 hours with S3 inventory report. That has always been good enough for me.

When you have billions of objects this is really the only way to go - and build workflows around inventory (including athena, spark, etc)

Re: S3 is files, but not a filesystem

#406

Post by EraYaN »

Kwpolska wrote: post #225
overstay8930 wrote: post #219

Earlier quoted context omitted.

We liked B2 but not enough to pay for IPv4 addresses, insane they advertise as a multi-cloud solution but basically kill any chance at adoption when NAT gateways and IPv4 charges are everywhere. We would literally save money paying B2 bandwidth fees (high read low write) but not when being pushed through a NAT64 gateway, or paying an hourly charge just to be able to access B2.

How could they launch a cloud service like this and not have IPv6 in 2015? What other basic things did they cheap out on?

Most mayor cloud vendors are still not fully dual stack capable so it's not that surprising. And plenty of ISPs have barely started rollout, or even said they just wont.

Re: S3 is files, but not a filesystem

#407

Post by readyman »

calpaterson wrote: post #400
readyman wrote: post #300

Earlier quoted context omitted.

"Even though the file API handles all those concerns, but it doesn't expose them to you. A narrow interface handling a large number of concerns - that makes the unix file API a "deep" module." Both sentences here are incomplete, incoherent. I did not read past this point.

Thanks. I take your point on the first one and have corrected it (maybe you need to shift+F5 to bust your cache in order to see it). For the second one, what's your objection? That it's fragmentary?

I don't know what it's trying to say. Making it a complete sentence would be a good first step. Don't try fancy stuff like this unless showing off style is more important to you than communicating coherently. In technical writing, little is more important than clarity.

Re: S3 is files, but not a filesystem

#408

Post by kragen »

breckognize wrote: post #266
pclmulqdq wrote: post #253

Earlier quoted context omitted.

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us. As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and…

0. user error (deletion or overwriting a file they regret later, possibly much later)

-1. government, the historical cause of most data loss

1½. google canceling the product and deleting all the data, as they did with google+

Re: S3 is files, but not a filesystem

#409

Post by hn72774 »

albert_e wrote: post #234
hn72774 wrote: post #173

Earlier quoted context omitted.

S3 is better for large datasets. It's cheaper and handles large file sizes with ease. It has become a de-facto standard for distributed, data-intensive workloads like those common with spark. A key benefit is decoupling the data from the compute so that they can scale independently. EBS is tightly coupled to iops and you pay extra for that. (Source: a long time working in data engineering)

Yes and I also believe: Experienced Spark / Data Engineering teams would not assume S3 is readily useable as a filesystem. This [1] seems like a good guide on how to configure spark for working with Cloud object stores, while recognizing the limitations and pitfalls. [1]: https://spark.apache.org/docs/latest/cloud-integration.html --- Amazon EMR offers a managed way to run hadoop or spark clusters and it implements a…

I think we're talking about two different things. I was addressing a section in the article about running databases backed by s3. It's less about s3 needing to act as a filesystem, and more about all of the rdbms features that come along with the various types of DB transactions. It's a solved problem with the libraries I mentioned. Not something I'd ever recommend to build on your own. Been there done that when those solutions were still nascent. Wasn't worth the effort vs just using an rdbms.

The problem that emrfs is trying to solve doesn't cover the rdbms scenarios like row-level updates and deletes.

Post reply on HN