Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Files

allthingsdistributed.com

Post reply on HN
Print view View original

101–110 of 128 posts

Re: S3 Files

#101

Post by vbezhenar »

wolttam wrote: post #64
dabinat wrote: post #62

The problem with using S3 as a filesystem is that it’s immutable, and that hasn’t changed with S3 Files. So if I have a large file and change 1 byte of it, or even just rename it, it needs to upload the entire file all over again. This seems most useful for read-heavy workflows of files that are small enough to fit in the cache.

That’s not that different than CoW filesystems - there is no rule that files must map 1:1 to objects; you can (transparently) divide a file into smaller chunks to enable more fine grained edits.

The most obvious approach seems to implement device blocks as S3 objects and use any existing file system on top of it.

Re: S3 Files

#103

Post by valyala »

This is a good alternative for those who wants storing petabytes of historical logs, metrics or traces in VictoriaLogs, VictoriaMetrics and VictoriaTraces, and wants saving 2x-4x on the persistent storage pricing (compare EBS pricing to S3 pricing).

Re: S3 Files

#104

Post by jollyllama »

gonzalohm wrote: post #13

I cannot 100% confirm this, but I believe AWS insisted a lot in NOT using S3 as a file system. Why the change now?

So that vibe-coders who don't understand s3 but have a user-level understanding of files can build stuff and pay them.

Re: S3 Files

#105

Post by tracerbits »

The interesting part isn't the file abstraction itself, it's that this pushes the dividing line between "object store" and "filesystem" another notch toward filesystem. The absence of in-place updates was always the load-bearing wall keeping S3 cheap and durable in the way it is — if Files preserves that and only makes the read API friendlier, fine.

If they ever ship in-place writes I'd want to see what happens to the consistency model and pricing first. That's where the actual simplicity lived, not in the API surface. Half the appeal of S3 over a real filesystem was that you couldn't shoot yourself in the foot with partial overwrites.

Re: S3 Files

#106

Post by harshaw »

this post will probably never be read but.. I was on the team that was trying to make the marriage of S3 and EFS work a year ago. it's a pretty hard problem. At one point we proposed this solution (which seems like a caching layer) but it got shot down for a more complex system that would have attempted to rebuild EFS on faster S3 blob storage. I left before this engineering monstrosity made significant progress, and it clearly died at some point.

Looks like they went back to a simpler solution they could deliver but with some obvious warts. good to see something get launched but the sausage making her was brutal.

The reality is that if you read https://www.allthingsdistributed.com/2026/04/s3-files-and-th..., it sounds like the great minds at S3 figured out that a caching layer was the way to go. We (EFS) fucking proposed that years ago. But we had to deal with Seattle and the S3 braintrust who didn't want to do that. I know we wrote a PRFAQ that was close to this concept probably four years ago. The political story is that EFS was taking over by S3 and the EFS folks didn't have the agency or political backing to build a more workable solution. So we wasted a shit ton of time tackling something that was never going to work and many of the tenured EFS engineers left.

Re: S3 Files

#107

Post by mgdev »

MontyCarloHall wrote: post #52
the8472 wrote: post #50

Earlier quoted context omitted.

NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs. The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.

I agree, it's an oddly low threshold. The latency differential of NFS vs. S3 is a couple OOMs, so a threshold of ~10MB seems more appropriate to me. Perhaps it's set intentionally low to avoid racking up immense EFS bills? Setting it higher would effectively mean getting billed $0.03/GB for a huge fraction of reads, which is untenable for most people's applications.

Once upon a time S3 used to cache small objects in their keymap layer, which IIRC had a similar threshold. I assume whatever new caching layer they added is piggybacking that.

This keeps the new caching layer simple and take advantage of the existing caching. If they went any bigger they'd likely need to rearchitect parts of the keymap or underlying storage layer to accommodate, or else face unpredictable TCO.

Re: S3 Files

#109

Post by otterley »

wolttam wrote: post #64
dabinat wrote: post #62

The problem with using S3 as a filesystem is that it’s immutable, and that hasn’t changed with S3 Files. So if I have a large file and change 1 byte of it, or even just rename it, it needs to upload the entire file all over again. This seems most useful for read-heavy workflows of files that are small enough to fit in the cache.

That’s not that different than CoW filesystems - there is no rule that files must map 1:1 to objects; you can (transparently) divide a file into smaller chunks to enable more fine grained edits.

The unit of granularity for a CoW filesystem is a block, which is typically 4kB or smaller. The unit of granularity for S3 is the entire object or 5MB (minimum multipart upload size), whichever is smaller. The difference can be immense.

Re: S3 Files

#110

Post by otterley »

Eikon wrote: post #90
miguel_martin wrote: post #42

Dumb Q: what would happen if you used this to store a SQLite database? Would it just... work? My guess is this would only enable a read-replica and not backups as Litestream currently does?

SQLite works great with ZeroFS: https://github.com/Barre/ZeroFS

Via which front end? It can’t be the NFS one.
Post reply on HN