S3 Files • Hacker News Board

Re: S3 Files

#91

Post by Eikon » Wed, Apr 08, 2026, 6:43 AM UTC

ChocolateGod wrote: post #23

huntaub wrote: post #19

Earlier quoted context omitted.

This is pretty different than s3fs. s3fs is a FUSE file system that is backed by S3. This means that all of the non-atomic operations that you might want to do on S3 (including edits to the middle of files, renames, etc) are run on the machine running S3fs. As a result, if your machine crashes, it's not clear what's going to show up in your S3 bucket or if would corrupt things. As a result, S3fs is also slow because…

You can also use something like JuiceFS to make using S3 as a shared filesystem more sane, but you're moving all the metadata to a shared database.

Or ZeroFS which doesn’t require a 3rd party database, just a s3 bucket!

https://github.com/Barre/ZeroFS

Re: S3 Files

#92

Post by Maxious » Wed, Apr 08, 2026, 7:29 AM UTC

PunchyHamster wrote: post #25

Eagerly awaiting on first blogpost where developers didn't read the eventually consistent part, lost the data and made some "genius" workaround with help of the LLM that got them in that spot in the first place

> Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent.

https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

Re: S3 Files

#93

Post by direwolf20 » Wed, Apr 08, 2026, 7:37 AM UTC

wolttam wrote: post #64

dabinat wrote: post #62

The problem with using S3 as a filesystem is that it’s immutable, and that hasn’t changed with S3 Files. So if I have a large file and change 1 byte of it, or even just rename it, it needs to upload the entire file all over again. This seems most useful for read-heavy workflows of files that are small enough to fit in the cache.

That’s not that different than CoW filesystems - there is no rule that files must map 1:1 to objects; you can (transparently) divide a file into smaller chunks to enable more fine grained edits.

But this doesn't

Re: S3 Files

#94

Post by WilcoKruijer » Wed, Apr 08, 2026, 7:56 AM UTC

rdtsc wrote: post #27

Synchronization bits is what I was wondering about: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-fil... > For example, suppose you edit /mnt/s3files/report.csv through the file system. Before S3 Files synchronizes your changes back to the S3 bucket, another application uploads a new version of report.csv directly to the S3 bucket. When S3 Files detects the conflict, it moves your version of report.csv to…

Mounting S3 buckets seemed like a great way to make stateless applications stateful for a while, which sounds appealing, especially for agent-like workloads. Handling conflicts like this means you really have to approach the mounted bucket as separate stateful thing. Seems like a mismatch to me.

Re: S3 Files

#95

Post by huksley » Wed, Apr 08, 2026, 8:56 AM UTC

I was prototyping with S3 mounted as filesystem for docker volumes and evaluating solutions for that. GeeseFS cli is the fastest one, here I made a script to mount folder with it from S3 compatible storage:

https://gist.github.com/huksley/44341276d7c269f092e10784959e...

You might want to play with memory params for GeeseFS for better results

Re: S3 Files

#96

Post by mememememememo » Wed, Apr 08, 2026, 10:15 AM UTC

boulos wrote: post #63

wbl wrote: post #39

"NFS provides the semantics your applications expect" is one of the funniest things I have ever read.

Compared to roll-your-own with S3 or GCS it does :)

Semanticness as a measurement.

Re: S3 Files

#97

Post by huntaub » Wed, Apr 08, 2026, 10:19 AM UTC

curt15 wrote: post #65

How does this compare with ZFS's object storage backend? https://news.ycombinator.com/item?id=46620673

Notably, this is going to manage your data in it's native format (i.e. you can actually read-write the files out of the S3 bucket as if they were actual objects, mapping 1:1 to each file). The ZFS backend is (almost certainly) a block-based format that is persisted to S3 (meaning that you cannot use it for existing data in S3, and you cannot access data written through ZFS via S3).

Re: S3 Files

#98

Post by huntaub » Wed, Apr 08, 2026, 10:20 AM UTC

mgaunard wrote: post #89

huntaub wrote: post #19

Earlier quoted context omitted.

This is pretty different than s3fs. s3fs is a FUSE file system that is backed by S3. This means that all of the non-atomic operations that you might want to do on S3 (including edits to the middle of files, renames, etc) are run on the machine running S3fs. As a result, if your machine crashes, it's not clear what's going to show up in your S3 bucket or if would corrupt things. As a result, S3fs is also slow because…

It also means that you need to pay for EFS, which is outrageously expensive, to use S3, whose whole purpose is to be cheap.

Of course, you don't need to, this is just a way to opt-in to getting file semantics on top of S3.

The purpose of S3 isn't to be cheap, it's to be simple.

Re: S3 Files

#99

Post by huntaub » Wed, Apr 08, 2026, 10:21 AM UTC

wenc wrote: post #59

hiyer wrote: post #44

Earlier quoted context omitted.

I was thinking of using it with Duckdb as well but seems it would be of limited benefit. Parquet objects are in MBs, so they would be streamed directly from S3. With raw parquet objects, it might help with S3 listing if you have a lot of them (shave off a couple of seconds from the query). If you are already on Ducklake, Duckdb will use that for getting the list of relevant objects anyway.

Maybe the OP is thinking of reading/writing to DuckDB native format files. Those require filesystem semantics for writing. Unfortunately, even NFS or SMB are not sufficiently FS-like for DuckDB. Parquet is static append only, so DuckDB has no problems with those living on S3.

What does DuckDB need that NFS/SMB do not provide?

Re: S3 Files

#100

Post by huntaub » Wed, Apr 08, 2026, 10:24 AM UTC

the8472 wrote: post #40

MontyCarloHall wrote: post #26

This is essentially S3FS using EFS (AWS's managed NFS service) as a cache layer for active data and small random accesses. Unfortunately, this also means that it comes with some of EFS's eye-watering pricing: — All writes cost $0.06/GB, since everything is first written to the EFS cache. For write-heavy applications, this could be a dealbreaker. — Reads hitting the cache get billed at $0.03/GB. Large reads (>128kB) g…

> Large reads (>128kB) get directly streamed from the underlying S3 bucket, which is free. Always uncached? S3 has pretty bad latency.

I imagine (hope) that they are doing some kind of intelligent read-ahead in the frontend servers to optimize for sequential reads that would avoid this looking terrible for applications.