S3 is files, but not a filesystem • Hacker News Board

Re: S3 is files, but not a filesystem

#171

Post by crabbone » Sun, Mar 10, 2024, 2:51 PM UTC

cuno wrote: post #162

calpaterson wrote: post #137

Earlier quoted context omitted.

I have to say that I'm not hugely convinced. I don't really think that being able to pull out the keys before or after a prefix is particularly impressive. That is the basis for database indices going back to the 1970s after all. Perhaps the use-cases you're talking about are very different from mine. That's possible of course. But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn'…

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

> find it faster than a local filesystem for many benchmarks.

What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work...

Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have?

----

The thing is: you should expect, in the best case, something like 5 ms latency on network calls over the Internet in an ideal case. Within the datacenter, maybe you can achieve sub-ms latency, but that's hard. AWS within region but different zones tends to be around 1 ms latency.

This is while NVMe latency, even on consumer products, is 10-20 micro seconds. I.e. we are talking about roughly 100 times faster than anything going through the network can offer.

Re: S3 is files, but not a filesystem

#172

Post by cuno » Sun, Mar 10, 2024, 3:00 PM UTC

crabbone wrote: post #171

cuno wrote: post #162

Earlier quoted context omitted.

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

> find it faster than a local filesystem for many benchmarks. What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work... Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have? ---- The thi…

For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/

and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...

Re: S3 is files, but not a filesystem

#173

Post by hn72774 » Sun, Mar 10, 2024, 3:08 PM UTC

albert_e wrote: post #58

hn72774 wrote: post #55

> Filesystem software, especially databases, can't be ported to Amazon S3 Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it. Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.

There is specifically block storage service (EBS) and falvirs of it like EBS multi-attach and EFS that can ne used if there is a need to port software/databases to the cloud with low level filesystem support. Why would we need to do it on object storage which addresses a different type of storage need. Nevertheless there are projects like EMRFS and S3 file system mount points that try to provide files stem interfaces…

S3 is better for large datasets. It's cheaper and handles large file sizes with ease.

It has become a de-facto standard for distributed, data-intensive workloads like those common with spark.

A key benefit is decoupling the data from the compute so that they can scale independently. EBS is tightly coupled to iops and you pay extra for that.

(Source: a long time working in data engineering)

Re: S3 is files, but not a filesystem

#174

Post by kbolino » Sun, Mar 10, 2024, 3:13 PM UTC

tjoff wrote: post #140

orf wrote: post #133

Earlier quoted context omitted.

Sure. It's kind of an index - limited to prefix-only searching, but useful. Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`. If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get…

But surely you need to track that elsewhere anyway? That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.

> But surely you need to track that elsewhere anyway?

Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else.

Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional services are unnecessary and keeping them in sync with S3 is more hassle than it's worth.

Re: S3 is files, but not a filesystem

#175

Post by supriyo-biswas » Sun, Mar 10, 2024, 3:13 PM UTC

cuno wrote: post #162

calpaterson wrote: post #137

Earlier quoted context omitted.

I have to say that I'm not hugely convinced. I don't really think that being able to pull out the keys before or after a prefix is particularly impressive. That is the basis for database indices going back to the 1970s after all. Perhaps the use-cases you're talking about are very different from mine. That's possible of course. But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn'…

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

> Once you start parallelising the ListObjectV2 (rather than sequentially "continuing")

How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.

Re: S3 is files, but not a filesystem

#176

Post by jugg1es » Sun, Mar 10, 2024, 3:14 PM UTC

hiAndrewQuinn wrote: post #57

I feel like I understand the lasting popularity of the humble FTP fileserver a bit better now. Thank you.

oh but amazon offers SFTP on top of S3 so you don't have to miss out.

Re: S3 is files, but not a filesystem

#177

Post by chrisblackwell » Sun, Mar 10, 2024, 3:20 PM UTC

Random note: Has anyone noticed how fast the author's webpage is? I know it's static, but I mean it's fast even for the DNS lookup. I would love to know what they have on.

Re: S3 is files, but not a filesystem

#178

Post by cuno » Sun, Mar 10, 2024, 3:20 PM UTC

supriyo-biswas wrote: post #175

cuno wrote: post #162

Earlier quoted context omitted.

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

> Once you start parallelising the ListObjectV2 (rather than sequentially "continuing") How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.

For example, you can do separate parallel ListObjectV2 for files starting a-f and g-k, etc.. covering the whole key space. You can parallelize recursively based on what is found in the first 1000 entries so that it matches the statistics of the keys. Yes there may be pathological cases, but in practice we find this works very well.

Re: S3 is files, but not a filesystem

#179

Post by radlad » Sun, Mar 10, 2024, 3:21 PM UTC

supriyo-biswas wrote: post #175

cuno wrote: post #162

Earlier quoted context omitted.

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

> Once you start parallelising the ListObjectV2 (rather than sequentially "continuing") How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.

You're right that it won't work for all use cases, but starting two threads with prefixes A and M, for example, is one way you might achieve this.

Re: S3 is files, but not a filesystem

#180

Post by alphazard » Sun, Mar 10, 2024, 3:22 PM UTC

S3 is not even files, and definitely not a filesystem.

The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I need to make a copy and re-upload. As originally conceived, the file abstraction finds some sectors on disk, and presents them to the client as a contiguous buffer. S3 solves a different problem.

Many people misinterpret the Good Idea from UNIX "everything is a file" to mean that everything should look like a contiguous virtual buffer. That's not what the real Good Idea is. Really: everything can be listed in a directory, including directories. There will be base leaves, which could be files, or any object the system wants to present to a process, and there will be recursive trees (which are directories). The directories are what make the filesystem, not the type of a particular leaf. Adding a new type of leaf, like a socket or a frame buffer, or whatever, is almost boring, and doesn't erode the integrity of the real good idea. Adding a different kind of container like a list, would make the structure of the filesystem more complex, and that would erode the conceptual integrity.

S3 doesn't do any of these things, and that's fine. I just want a place to put things that won't fit in the database, and know they won't bitrot when I'm not looking. The desire to make S3 look more like a filesystem comes from client misunderstanding of what it's good at/for, and poor product management indulging that misunderstanding instead of guarding the system from it.