121–128 of 128 posts
Earlier quoted context omitted.
Reads from s3 via an s3 endpoint inside a vpc to an interface inside of that vpc is not billed.
S3 GET operations are billed anyway. Traffic may be free, but not the operations.
Earlier quoted context omitted.
Reads from s3 via an s3 endpoint inside a vpc to an interface inside of that vpc is not billed.
S3 GET operations are billed anyway. Traffic may be free, but not the operations.
Earlier quoted context omitted.
That’s not that different than CoW filesystems - there is no rule that files must map 1:1 to objects; you can (transparently) divide a file into smaller chunks to enable more fine grained edits.
The most obvious approach seems to implement device blocks as S3 objects and use any existing file system on top of it.
Earlier quoted context omitted.
S3 GET operations are billed anyway. Traffic may be free, but not the operations.
Ok, but to be clear you said all outgoing traffic is charged. The API call price is something else, it's not part of the outgoing traffic.
Your example with VPC endpoints does not leave AWS.
Sorry for the confusion you probably thought about outgoing from S3, but I mean outgoing from AWS.
Earlier quoted context omitted.
S3 GET operations are billed anyway. Traffic may be free, but not the operations.
S3 GET requests have negligible cost, at $0.0004/1000 requests. Upload requests are $0.005/1000, which is also negligible.
Earlier quoted context omitted.
You can also use something like JuiceFS to make using S3 as a shared filesystem more sane, but you're moving all the metadata to a shared database.
Or ZeroFS which doesn’t require a 3rd party database, just a s3 bucket! https://github.com/Barre/ZeroFS
Earlier quoted context omitted.
I am curious about this use case How do you see it helping with DuckLake?
Latency, predicate pushdown. Pre-compaction the recent data can be in small files, and the delete markers will also be in small files. This will bring down fetch times, while ducklake may have many of the larger blocks in memory or disk cache already. Reading block headers for filtering is lots of small ranges, this could speed it up by 10x.
Would be really useful pre-compaction and to deal with small files issue without latency penalties