S3 is files, but not a filesystem • Hacker News Board

Re: S3 is files, but not a filesystem

#241

Post by somedudetbh » Sun, Mar 10, 2024, 5:24 PM UTC

> Amazon S3 is the original cloud technology: it came out in 2006. "Objects" were popular at the time and S3 was labelled an "object store", but everyone really knows that S3 is for files. S3

Alternative theory: everyone who worked on this knew that it was not a filesystem and "object store" is a description intended to describe everything else pointed out in this post.

"Objects were really popular" is about objects as software component that combines executable code with local state. None of the original S3 examples were about "hey you can serialize live objects to this store and then deserialize them into another live process!" It was all like "hey you know how you have all those static assets for your website..." "Objects" was used in this sense in databases at the time in the phrase "binary large object" or "blob". S3 was like "hey, stuff that doesn't fit in your database, you know...objects...this is a store for them."

This is meant to describe precisely things like "listing is slow" because when S3 was designed, the launch usecases assumed an index of contents existed _somewhere else_, because, yeah, it's not a filesystem. it's an object store.

Re: S3 is files, but not a filesystem

#242

Post by allset_ » Sun, Mar 10, 2024, 5:27 PM UTC

rsync wrote: post #206

breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

"AWS' availability zone isolation is better than the other cloud providers." Not better than all of them. A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver. "S3 even operates at a scale where we could detect "bitrot"" That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale invol…

FWIW, both AWS S3 and GCP GCS also allow you to store data in multi-region.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiR...

https://cloud.google.com/storage/docs/locations#consideratio...

Re: S3 is files, but not a filesystem

#243

Post by tutfbhuf » Sun, Mar 10, 2024, 5:29 PM UTC

S3 is obviously not a filesystem in the sense of a POSIX filesystem. And I would argue it is not a filesystem, even if we were to relax POSIX filesystem semantics (do not implement the full spec). But what is certainly possible is to span a filesystem on top of S3. It is basically possible to span a filesystem on anything that can store data. You can even go crazy for demonstration purposes and put a filesystem on top of YouTube (there are some tech demos for that on GitHub).

I think a better question is whether there are any good filesystem implementations on top of S3. There are many attempts like s3fs-fuse[^1] or seaweedfs[^2], but I have not heard many stories about their use at scale from big companies. Just recently there was a post here about cunoFS[^3]. It is a startup that implements a POSIX-compliant (supports symlinks, hard links (emulated), UIDs & GIDs, permissions, random writes, etc.) filesystem on top of S3/AZ/GCP storage and claims to have really good performance. I think only time will tell if it works out in practice for companies to use S3 as a filesystem through fs implementations on top of S3.

[^1]: https://github.com/s3fs-fuse/s3fs-fuse

[^2]: https://github.com/seaweedfs/seaweedfs

[^3]: https://news.ycombinator.com/item?id=39640307

Re: S3 is files, but not a filesystem

#244

Post by markhahn » Sun, Mar 10, 2024, 5:29 PM UTC

MatthiasPortzel wrote: post #159

This article was an epiphany for me because I realized I've been thinking of the Unix filesystem as if it has two functions: read_file and write_file. (And then getting frustrated with the filesystem APIs in programming languages.)

So you came from an S3 or other put-get world, and found actual filesystems odd?

I suppose that's not so different from a WMP user's epiphany when they discover processes, shells, etc.

Re: S3 is files, but not a filesystem

#245

Post by thinkharderdev » Sun, Mar 10, 2024, 5:30 PM UTC

alphazard wrote: post #180

S3 is not even files, and definitely not a filesystem. The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I ne…

> S3 is not even files, and definitely not a filesystem.

I agree. To me the correct analog for S3 is a block storage device (a very weird one where blocks can be any size and can have a key associated with them) and not a filesystem. A filesystem is an abstraction that sits on top of a block storage device and so an "S3 filesystem" would have to be an abstraction that sits on top of S3 as the underlying block storage.

Re: S3 is files, but not a filesystem

#246

Post by emodendroket » Sun, Mar 10, 2024, 5:31 PM UTC

logifail wrote: post #240

breckognize wrote: post #216

Earlier quoted context omitted.

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver. The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough tha…

> The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle What may be less of a sweet spot is AWS' pricing.

Sending the data to /dev/null is the cheapest option if that’s all you care about.

Re: S3 is files, but not a filesystem

#247

Post by jamesrat » Sun, Mar 10, 2024, 5:34 PM UTC

donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

I implemented a solution by threading the listing. Get the files in the root then spin a separate process to do the recursion for each directory.

Re: S3 is files, but not a filesystem

#248

Post by wenc » Sun, Mar 10, 2024, 5:35 PM UTC

cuno wrote: post #172

crabbone wrote: post #171

Earlier quoted context omitted.

> find it faster than a local filesystem for many benchmarks. What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work... Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have? ---- The thi…

For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/ and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...

S3 is really high latency though. I store parquet files on S3 and querying them through DuckDB is much slower than file system because random access patterns. I can see S3 being decent if it’s bulk access but definitely not for random access.

This is why there’s a new S3 Express offering that is low latency (but costs more).

Re: S3 is files, but not a filesystem

#249

Post by teaearlgraycold » Sun, Mar 10, 2024, 5:36 PM UTC

d-z-m wrote: post #210

> S3 is a cloud filesystem, not an object-whatever. [...]I think the idea that S3 is really "Amazon Cloud Filesystem" is a bit of a load bearing fiction. Does anyone actually think this? I have never encountered anyone who has described S3 in these terms.

Not sure if the author is aware of EFS

Re: S3 is files, but not a filesystem

#250

Post by seabrookmx » Sun, Mar 10, 2024, 5:37 PM UTC

BirAdam wrote: post #232

Underneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.

S3 actually uses a completely custom system[1] for writing bytes to disk. I haven't seen much in the way of details on the on-disk format but I certainly wouldn't assume it resembles a normal filesystem.

[1]: https://aws.amazon.com/blogs/storage/how-automated-reasoning...