Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Files

allthingsdistributed.com

Post reply on HN
Print view View original

111–120 of 128 posts

Re: S3 Files

#111

Post by rufo »

harshaw wrote: post #106

this post will probably never be read but.. I was on the team that was trying to make the marriage of S3 and EFS work a year ago. it's a pretty hard problem. At one point we proposed this solution (which seems like a caching layer) but it got shot down for a more complex system that would have attempted to rebuild EFS on faster S3 blob storage. I left before this engineering monstrosity made significant progress, and…

I don’t have much actively constructive to say, but having worked in a large engineering organization before - boy, do I feel this.

Re: S3 Files

#112

Post by jlokier »

laurencerowe wrote: post #56
miguel_martin wrote: post #42

Dumb Q: what would happen if you used this to store a SQLite database? Would it just... work? My guess is this would only enable a read-replica and not backups as Litestream currently does?

SQLite’s locking is not NFS safe so this would not work.

Technically, SQLite's locking is NFS safe, provided NFS's implementation of fcntl() locking is working correctly.

I don't know if S3 Files implements fcntl() locking or does it correctly. But if it does, I believe SQLite should work on it correctly as well.

There have been many buggy NFS locking or caching implementations historically, which is why reason SQLite recommends against using it on NFS concurrently on multiple machines: https://sqlite.org/faq.html#:~:text=But%20use,time%2E

This SO reply suggests NFSv4 is better at this: https://unix.stackexchange.com/a/432519. But caveat it with this older reply: https://unix.stackexchange.com/a/1887

To the best of my knowledge (I worked a little on this long ago), on Linux even NFSv2 has done correct fcntl() locking for decades, if all the correct services are running and the options are set appropriately and it's Linux on both the client and server. But if something is not configured as it should be, then locking or caching may not work correctly.

Re: S3 Files

#113

Post by objectivefs »

MontyCarloHall wrote: post #26

This is essentially S3FS using EFS (AWS's managed NFS service) as a cache layer for active data and small random accesses. Unfortunately, this also means that it comes with some of EFS's eye-watering pricing: — All writes cost $0.06/GB, since everything is first written to the EFS cache. For write-heavy applications, this could be a dealbreaker. — Reads hitting the cache get billed at $0.03/GB. Large reads (>128kB) g…

One advantage over S3FS would be that multiple filesystem mounts would see a consistent view of the filesystem, but it looks like this advantage disappears when mixing direct bucket access with filesystem mounts. Given the famously slow small file performance of EFS it might have been better (and cheaper) to send all files to S3 and only use EFS for the metadata layer. Not having atomic rename is also going to be a problem for any use that expects a regular filesystem.

Re: S3 Files

#114

Post by jamesblonde »

harshaw wrote: post #106

this post will probably never be read but.. I was on the team that was trying to make the marriage of S3 and EFS work a year ago. it's a pretty hard problem. At one point we proposed this solution (which seems like a caching layer) but it got shot down for a more complex system that would have attempted to rebuild EFS on faster S3 blob storage. I left before this engineering monstrosity made significant progress, and…

How could they release something that doesn't support atomic rename and has no prospect of supporting atomic rename? Lots of workloads will crash and burn on this layer.

Re: S3 Files

#115

Post by laurencerowe »

jlokier wrote: post #112
laurencerowe wrote: post #56

Earlier quoted context omitted.

SQLite’s locking is not NFS safe so this would not work.

Technically, SQLite's locking is NFS safe, provided NFS's implementation of fcntl() locking is working correctly. I don't know if S3 Files implements fcntl() locking or does it correctly. But if it does, I believe SQLite should work on it correctly as well. There have been many buggy NFS locking or caching implementations historically, which is why reason SQLite recommends against using it on NFS concurrently on mult…

Thanks for the clarification. It is completely impossible for WAL mode since that uses shared memory. I must have conflated that with non-WAL mode in my mind.

From https://sqlite.org/wal.html

> All processes using a database must be on the same host computer; WAL does not work over a network filesystem. This is because WAL requires all processes to share a small amount of memory and processes on separate host machines obviously cannot share memory with each other.

Re: S3 Files

#116

Post by borplk »

Does anyone have solutions or suggestions for mounting a S3 bucket as a read-only filesystem? I don't need any writes.

Previously I have done a periodic script that would simply re-sync the directory which works well enough. But curious if there's anything else out there.

Re: S3 Files

#117

Post by deepsun »

simtel20 wrote: post #84
deepsun wrote: post #75

Earlier quoted context omitted.

> directly streamed from the underlying S3 bucket, which is free. No reads from S3 are free. All outgoing traffic from AWS is charged no matter what.

Reads from s3 via an s3 endpoint inside a vpc to an interface inside of that vpc is not billed.

S3 GET operations are billed anyway.

Traffic may be free, but not the operations.

Re: S3 Files

#118

Post by pvtmert »

harshaw wrote: post #106

this post will probably never be read but.. I was on the team that was trying to make the marriage of S3 and EFS work a year ago. it's a pretty hard problem. At one point we proposed this solution (which seems like a caching layer) but it got shot down for a more complex system that would have attempted to rebuild EFS on faster S3 blob storage. I left before this engineering monstrosity made significant progress, and…

100% agree to this sentiment. Although Amazon/AWS seems to be overall innovative, amount of ideas and passion killed between the same meeting rooms the article describes as "innovating driving, heated conversations" are immense.

Obviously not the same, but at home I am running a Raspberry Pi with s3fs mounting my personal S3 bucket. I am exposing the same directory with /etc/exports (NFS). Which also allows me to use filesystem-caching as a bonus on the client side.

On the other hand, I should probably move out from S3 and use R2 or something...

Re: S3 Files

#119

Post by ktimespi »

avereveard wrote: post #85
ktimespi wrote: post #70

Earlier quoted context omitted.

This was my concern too. The whole point of using S3 as a file system instead of EBS / EFS (for me at least) is to minimize cost and I don't really see why I would use this instead of s3fs.

Probably some tradeoff at high client count or if you seek into files to read partial data

s3fs can do partial reads too with range queries, I'm leaning more towards the tradeoff.

Re: S3 Files

#120

Post by sb-gcs »

jamesblonde wrote: post #77

S3 Files was launched today without support for atomic rename. This is not something you can bolt on. Can you imagine running Claude Code on your S3 Files and it just wants to do a little house cleaning, renaming a directory and suddenly a full copy is needed for every file in that directory? The hardest part in building a distributed filesystem is atomic rename. It's always rename. Scalable metadata file systems, li…

Hierarchical Namespace buckets in Google Cloud Storage support folder operations, including atomic folder renames.

https://docs.cloud.google.com/storage/docs/hns-overview#feat...

Post reply on HN