Live data from Hacker News

Amazon Glacier

aws.amazon.com

181–190 of 393 posts

Re: Amazon Glacier

#181

Earlier quoted context omitted.

US Standard isn't replicated across regions.

Hm, you could be right, AWS says: > The US Standard Region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps. I guess this means that the data is automatically geographically sharded?

Keep in mind that even if your data is in one AWS region, it'll still be stored in multiple different datacenters some distance apart. Just not on the other side of the US.

Re: Amazon Glacier

#182
post #69

Earlier quoted context omitted.

sigh Dropbox should not be used as a backup system. A system that synchronizes live should never get that role, except if they can guarantee that old data is never overwritten, new is always appended. This is not the case with dropbox - I've experienced multiple scary occurrences of old versions going to the nirvana with certain user actions. In certain cases the old data just appears to be gone, in other cases the w…

I use a combination of Dropbox and S3. I have my MacBook and a Linux server linked to my Dropbox account. So changes in my documents are synced to my Linux. My Linux run three cron-jobs. One daily, one weekly and another monthly. The command is. s3cmd sync --delete-removed ~/Dropbox/documents/ s3://backup-daily/ There are buckets for weekly and monthly too. Note: the command is not exactly like that, check the man pa…

Wait, do I understand this correctly: You sync your dropbox to S3? Do you overwrite your previous daily when you do so?

Re: Amazon Glacier

#183
post #75

Anyone know what the TOS are for this? I couldn't find them on a scan of the announcement. A lot of the consumer-level services refuse any liability for any data loss. Does Amazon do the same for this?

Presumably the standard S3 SLA will apply: http://aws.amazon.com/s3-sla/ Realistically, you'd want to have at least two diverse cloud backup systems - I doubt you'd be happy with Service Credits if your data went missing.

It always seem funny though that these companies say

"We will keep your data safe! *

* T&C's apply, if we lose it, you're on your own."

I shouldn't imagine Bank Vaults deal the same way with physical property. Can you get insurance for digital assets the same way you can for physical ones?

Re: Amazon Glacier

#184

Earlier quoted context omitted.

With large volumes the real issue is not the storage but the upload speed. I've done some experiments and with the Comcast link (20Mbps down/whatever up), I got 1 Gb/hour upload rate. So, it'll take 11 full days to upload 256 Gb. Or, more realistically, if you do it overnight (8 hours) - the entire month.

If you don't mind the additional cost, you can physically send Amazon a harddrive: http://aws.amazon.com/importexport/

nothing says "cloud" like sending a physical hard drive through the mail.

Re: Amazon Glacier

#185

I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage. With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,v…

I'm looking into supporting Glacier in Arq. It sure is cheap -- $10/month for a terabyte.

That would be fantastic.

Re: Amazon Glacier

#186
post #98

Earlier quoted context omitted.

Duplciity supports S3, so I'd watch it for Glacier support: http://duplicity.nongnu.org/

I wonder if Glacier support in Duplicity will be possible without large changes. AFAIK, duplicity also reads some state from the remote end to determine what to backup (Although it also keeps a local cache of this?). To use glacier, the protocol would have to completely write-only.

I'd guess it would use a hybrid approach, with recent backups on S3 (which duplicity already does) being shifted to glacier after a period of time. The FAQ indicates that Amazon plans to make this easy.

Re: Amazon Glacier

#188
post #162

Earlier quoted context omitted.

In a corporate environment, I wouldn't want to depend on the cloud as my primary backup solution in the first place. I'd be much more comfortable using it as the offsite mirror of an onsite backup. If you're at a point in disaster recovery where you have to restore from your offsite, you (likely) have bigger problems than a 4-hour wait time.

I personally believe that data should never be deleted (or overwritten), but only appended to. Kinda like what redis/datomic does. So, keep live data onsite, along with an onsite (small) backup, and all the old data in Glacier.

You can believe that, but legal realities dictate otherwise. There are certain classes of information that you are not permitted to keep beyond a defined horizon, either temporal or event-based. Legal compliance with records management processes means having the ability to delete or destroy information such that it cannot be recovered. Note that if the information is encrypted, you can just delete the decryption key and it is effectively deleted.

Re: Amazon Glacier

#189
post #61
post #7

I had a quick skim through the marketing stuff and the FAQs and didn't see anywhere that actually details what the backend of this is. I'd be curious if they're actually using tape, older machines, Backblaze pods, etc. I guess if it's the latter, the time to recover could be an artificial barrier to prevent people from getting cute.

agreed, it would be great to know how this is running from a hardware point of view - just out of personal interest :-)

Someone further up mentioned a very plausible (in my experience) answer. Magnetic tape, using hard drive arrays as RAM. The wait time in this situation would be the time needed to complete all the current tasks waiting to be written/read in the queue before your data is written from tape to hard drive so you can access it.

Re: Amazon Glacier

#190

I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage. With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,v…

With large volumes the real issue is not the storage but the upload speed. I've done some experiments and with the Comcast link (20Mbps down/whatever up), I got 1 Gb/hour upload rate. So, it'll take 11 full days to upload 256 Gb. Or, more realistically, if you do it overnight (8 hours) - the entire month.

Makes me wonder what Amazon is doing to commoditize its complements, i.e. what it's doing to improve high speed internet access. There are a lot of people in my vicinity who have no option faster than a wireless 3Mbps connection capped at a few GB per day. And this is within easy distance of Amazon's East Coast data center.
Post reply on HN