S3 Files • Hacker News Board

Re: S3 Files

#31

Post by jitl » Tue, Apr 07, 2026, 9:24 PM UTC

gonzalohm wrote: post #13

I cannot 100% confirm this, but I believe AWS insisted a lot in NOT using S3 as a file system. Why the change now?

Because without significant engineering effort (see the blog post), the mismatch between object store semantics and file semantics mean you will probably Have A Bad Time. In much earlier eras of S3, there were also some implementation specifics like throughput limits based on key prefixes (that one vanished circa 2016) that made it even worse to use for hierarchical directory shapes.

Re: S3 Files

#32

Post by nyc_pizzadev » Tue, Apr 07, 2026, 9:28 PM UTC

This is very close to its first official release: https://fiberfs.io/

Built in cache, CDN compatible, JSON metadata, concurrency safe and it targets all S3 compatible storage systems.

Re: S3 Files

#33

Post by jitl » Tue, Apr 07, 2026, 9:31 PM UTC

I wish they offered some managed bridging to local NVMe storage. AWS NVMe is super fast compared to EBS, and EBS (node-exclusive access as block device) is faster than EFS (multi-node access). I imagine this can go fast if you put some kind of further-cache-to-NVMe FS on top, but a completely vertically integrated option would be much better.

Re: S3 Files

#34

Post by mritchie712 » Tue, Apr 07, 2026, 9:35 PM UTC

tldr: this caches your S3 data in EFS.

we run datalakes using DuckLake and this sounds really useful. GCP should follow suit quickly.

Re: S3 Files

#35

Post by minutesmith » Tue, Apr 07, 2026, 9:41 PM UTC

[flagged]

Re: S3 Files

#36

Post by glenjamin » Tue, Apr 07, 2026, 9:47 PM UTC

minutesmith wrote: post #35

[flagged]

The way AWS keep their pricing section completely separate from their system and architecture docs, despite architecture being the primary driver of cost, is a major contributor to this

Re: S3 Files

#37

Post by up2isomorphism » Tue, Apr 07, 2026, 9:55 PM UTC

This why today’s sales pitch are often disguised as a tech blog.

Re: S3 Files

#38

Post by dang » Tue, Apr 07, 2026, 10:28 PM UTC

Since this is the thread that got attention, I've added the announcement link to the toptext and made the title work for both.

Re: S3 Files

#39

Post by wbl » Tue, Apr 07, 2026, 10:35 PM UTC

"NFS provides the semantics your applications expect" is one of the funniest things I have ever read.

Re: S3 Files

#40

Post by the8472 » Tue, Apr 07, 2026, 10:39 PM UTC

MontyCarloHall wrote: post #26

This is essentially S3FS using EFS (AWS's managed NFS service) as a cache layer for active data and small random accesses. Unfortunately, this also means that it comes with some of EFS's eye-watering pricing: — All writes cost $0.06/GB, since everything is first written to the EFS cache. For write-heavy applications, this could be a dealbreaker. — Reads hitting the cache get billed at $0.03/GB. Large reads (>128kB) g…

> Large reads (>128kB) get directly streamed from the underlying S3 bucket, which is free.

Always uncached? S3 has pretty bad latency.