Glacier Costlier Than S3 for Small Files
therub.org
Glacier Costlier Than S3 for Small Files
1–10 of 17 posts
Re: Glacier Costlier Than S3 for Small Files
#2Unfortunately, I learned it the hard way :(
Re: Glacier Costlier Than S3 for Small Files
#3The real question is why are people glaciering small files?
Tar everything and send it up. And then do an incremental backup
Amazon charges per operation, reduce the number of operations
Re: Glacier Costlier Than S3 for Small Files
#4The real question is why are people glaciering small files? Tar everything and send it up. And then do an incremental backup Amazon charges per operation, reduce the number of operations
My guess is they don't realise. AWS let's you setup lifecycle rules to archive S3 objects to Glacier automatically. It doesn't explicitly say it is cheaper (it says "this may reduce your costs"), but they probably just saw the headline prices for Glacier and moved everything.
Re: Glacier Costlier Than S3 for Small Files
#5AWS is more honest with S3 Infrequent Access:
https://aws.amazon.com/s3/storage-classes/
No additional metadata, but you get charged at least for 128 KB . Luckily lifecycle transition doesn't move files smaller than 128 KB. However, even using lifecycle transition you pay $0.01 per 1000 transitions. It doesn't seem much but for smaller items it can decrease savings a lot. E.g. if you average file is 6 MB, than you will loose 4 days of savings on S3 IA comparing to standard class.
Re: Glacier Costlier Than S3 for Small Files
#6> For consistency and precision, the following units are used throughout this article.
> KB: 1,024 bytes, expressed as 2^10
Why not just use KiB? Unlike KB, it's unambiguously binary.
Re: Glacier Costlier Than S3 for Small Files
#7The real question is why are people glaciering small files? Tar everything and send it up. And then do an incremental backup Amazon charges per operation, reduce the number of operations
My guess is they don't realise. AWS let's you setup lifecycle rules to archive S3 objects to Glacier automatically. It doesn't explicitly say it is cheaper (it says "this may reduce your costs"), but they probably just saw the headline prices for Glacier and moved everything.
Also, I would guess a lot of Glacier users are probably just slotting it in place of some other system that doesn't have the same pricing model.
Re: Glacier Costlier Than S3 for Small Files
#8Personally, I'm looking forward to Backblaze's cloud storage, which is being advertised at $0.005/GB/month. Cheaper than glacier, without all of the transaction fees.
Re: Glacier Costlier Than S3 for Small Files
#9AWS has a wizard for bucket storage policy where they tell you this information explicitly with a big warning sign.
Re: Glacier Costlier Than S3 for Small Files
#10This is one of the things I love about GCS Nearline, it really is a penny per byte and then the whole retrieval charges (which for my lazy rsync backup is never).
Disclaimer: I work at Google on Compute Engine, but not on GCS.