Any desktop apps out there that will let you add folders for backup to Glacier and have them be automatically synced up to the cloud as they change? That would be quite useful.
Amazon Glacier
71–80 of 393 posts
Re: Amazon Glacier
#72Storage experts: I'd love to know more about what might be backing this service. What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?
Typically they are tiered. There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access. Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array. Waiting for a slot with the robot arm - tape drive is what will take 4 hours. EMC(kinda), Fujitsu etc…
Thanks -- I had no idea you could just buy a tape library off the rack (more or less).
Re: Amazon Glacier
#73Earlier quoted context omitted.
Typically they are tiered. There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access. Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array. Waiting for a slot with the robot arm - tape drive is what will take 4 hours. EMC(kinda), Fujitsu etc…
Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.
The multiple-hour window could give you a lot of wiggle room here though. It's unlikely to take 3 hours to restore from a single tape, so even if you have to visit 2-3 tapes then you have plenty of time.
I'm sure that there is a general tiered storage platform (as mentioned above) which keeps some of the data online as well. That would let you run a "defrag" algorithm later if you find you need it.
Re: Amazon Glacier
#74that's certainly interesting. as there will be migration from s3 to glacier, it would be nice if tarsnap had an option to store only the (say) last week in s3 (with .3$/gb/month) and the rest in glacier (with, say, .03$/gb/month). that would certainly be very nice. cperciva, what do you think?
I can't see any way for Tarsnap to use this right now. When you create a new archive, you're only uploading new blocks of data; the server has no way of knowing which old blocks of data are being re-used. As a result, storing any significant portion of a user's data in Amazon Glacier would mean that all archive extracts would need to go out to Glacier for data... Also, with Tarsnap's average block size (~ 64 kB uncom…
While I have no idea how you would fit it in your current infrastructure, I certainly see a (BIG) use-case for: I have this 100 GB, store it somewhere safe (in glacier), I won't need it for the next year (unless my house burns down).
I agree that is a bit different from ongoing daily backups with changes, but its also not THAT different from a customer perspective.
That it doesn't fit with how you store blocks on the backend won't matter to a lot of customers.
Re: Amazon Glacier
#75Anyone know what the TOS are for this? I couldn't find them on a scan of the announcement.
A lot of the consumer-level services refuse any liability for any data loss. Does Amazon do the same for this?
Re: Amazon Glacier
#76Interestingly they penalise you for short-term storage:
Amazon Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months
Re: Amazon Glacier
#77This is fantastic. I've long searched for a solution like that. This is really suitable for a remote backup that only needs to be accessed if something really bad happens (i.e. a fire breaking out, etc). I'm a lone entrepreneur, so I do have backup hard disks here, but being able to additionally save this data in the cloud is great. I'm often creating pretty big media assets, so Dropbox doesn't necessarily offer enou…
The only issue I see is that verifying archive integrity (you don't want to find out the archive was bad after you lost the local backup...) would be somewhat complicated, given their retrieval policies. Also, the billing for data-transfer out plus peak retrievals sounds so convoluted, I can't begin to work out what a regular test-restore procedure would cost me. Nevertheless, it's some exciting progress in remote storage!
Re: Amazon Glacier
#78Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month. What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics. Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.
> Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot. As long as that data is decode-able and more importantly, find-able (out of all the GBs frozen for 100 years, why would you want to look at any particular one of them?).
To be fair, if improvements in hardware and software continue at the rate they have been, or some moderate percentage thereof, in 100 years it will be no problem to trawl a few exabytes of data for anything interesting.
Re: Amazon Glacier
#79Earlier quoted context omitted.
Typically they are tiered. There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access. Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array. Waiting for a slot with the robot arm - tape drive is what will take 4 hours. EMC(kinda), Fujitsu etc…
Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.
I'd guess that they ignore that problem and have baked the time it takes to get data from several tapes into the 3-4 hour estimate.
If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time reading. Also in backup situations like this, it's more important to get the backup done in a timely manner.
Re: Amazon Glacier
#80I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage. With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,v…
What's even more important, you will be able to encrypt your backups without having to disclose the encryption key in case you ever need to restore (client-side encryption and decryption). This is not the case with Backblaze, which is why I switched to CrashPlan — but I'm still looking for other solutions.