S3 is files, but not a filesystem • Hacker News Board

Re: S3 is files, but not a filesystem

#181

Post by hnlmorg » Sun, Mar 10, 2024, 3:22 PM UTC

cuno wrote: post #172

crabbone wrote: post #171

Earlier quoted context omitted.

> find it faster than a local filesystem for many benchmarks. What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work... Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have? ---- The thi…

For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/ and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...

EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.

Re: S3 is files, but not a filesystem

#182

Post by kbolino » Sun, Mar 10, 2024, 3:25 PM UTC

adrian_b wrote: post #119

orf wrote: post #115

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful . What makes it useful is listing. In an unversioned bucket (or one with no delete markers), li…

Since 30 years ago (starting with XFS in 1993, which was inspired by HPFS) all the good UNIX file systems implement the directories as some kind of B trees. Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.

Resolving random file system paths still gets slower proportional to their depth, which is not the case for S3, where the prefix is on the entire object key and not just the "basename" part of it, like in a filesystem.

Re: S3 is files, but not a filesystem

#183

Post by akerl_ » Sun, Mar 10, 2024, 3:27 PM UTC

alphazard wrote: post #180

S3 is not even files, and definitely not a filesystem. The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I ne…

How do read-only filesystems align with your definition?

Re: S3 is files, but not a filesystem

#184

Post by hiAndrewQuinn » Sun, Mar 10, 2024, 3:29 PM UTC

jugg1es wrote: post #176

hiAndrewQuinn wrote: post #57

I feel like I understand the lasting popularity of the humble FTP fileserver a bit better now. Thank you.

oh but amazon offers SFTP on top of S3 so you don't have to miss out.

If it's offered on top of S3, though, doesn't it still have all the same issues of needing to totally overwrite files?

Re: S3 is files, but not a filesystem

#185

Post by adverbly » Sun, Mar 10, 2024, 3:30 PM UTC

chrisblackwell wrote: post #177

Random note: Has anyone noticed how fast the author's webpage is? I know it's static, but I mean it's fast even for the DNS lookup. I would love to know what they have on.

The response headers include

server: cloudflare

You said it though - the reason is that its static without any js/frameworks/SPA round trip requests.

Re: S3 is files, but not a filesystem

#186

Post by _9hey » Sun, Mar 10, 2024, 3:31 PM UTC

It’s been a while, but I really like the way google handles its file system internally. No confusion.

Re: S3 is files, but not a filesystem

#187

Post by remram » Sun, Mar 10, 2024, 3:34 PM UTC

I am currently pondering this exact problem. I want to run a file-sharing web application (think: NextCloud) but I don't want to use expensive block storage or the dedicated server's disk space for the files, as some of them will be accessed infrequently.

I am wondering if s3fs/rclone-mount is sufficient, or if I should use something like JuiceFS that adds random-access, renaming, etc on top of it. Are those really necessary APIs for my use case? Is there only one way to find out?

(The app doesn't have native S3 support)

Re: S3 is files, but not a filesystem

#188

Post by jkoudys » Sun, Mar 10, 2024, 3:35 PM UTC

I absolutely loved this article. Super well written with interesting insights.

Re: S3 is files, but not a filesystem

#189

Post by donatj » Sun, Mar 10, 2024, 3:35 PM UTC

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem.

I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me his pre-populated cache. My initial thought was “this can’t really be necessary” and started poking.

We have ~100,000 root level directories for our individual assets. Each of those have five or six directories with a handful of files. Probably less than a million files total, maybe 3 levels deep at its deepest.

Recursively listing these files takes literally fifteen minutes. I poked and prodded suggestions from stack overflow and ChatGPT at potential ways to speed up the process and got nothing notable. That’s absurdly slow. Why on earth is it so slow?

Why is this something Amazon has not fixed? From the outside really seems like they could slap some B-trees on the individual buckets and call it a day.

If it is a difficult problem, I’m sure it would be for fascinating reasons I’d love to hear about.

Re: S3 is files, but not a filesystem

#190

Post by macintux » Sun, Mar 10, 2024, 3:35 PM UTC

belter wrote: post #156

tjoff wrote: post #128

Earlier quoted context omitted.

Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.

No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

I don't know that you can characterize that as a "standard practice".

Maybe it's widespread, but I've not encountered it.