Live data from Hacker News

Amazon Glacier

aws.amazon.com

201–210 of 393 posts

Re: Amazon Glacier

#201
post #184

Earlier quoted context omitted.

If you don't mind the additional cost, you can physically send Amazon a harddrive: http://aws.amazon.com/importexport/

nothing says "cloud" like sending a physical hard drive through the mail.

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S." - http://en.wikipedia.org/wiki/Sneakernet

Re: Amazon Glacier

#202
post #90

I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage. With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,v…

Home-use is probably the only situation Glacier is good for backup though. A home user is fine with a 3.5-4 hour window before their backup becomes available for download (as it will probably take them days to download it anyway). In a corporate environment, I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation. Seems good for archive-only in a…

It's made for archiving, not 'backups'. You're expected to keep most recent backups on-site, so if a RAID array dies or you happen to do something wrong, it just takes minutes to restore. Off-site, or tape archive, it's usually another thing. Most of the data is never, ever accessed (but required by company policy to be kept).

Re: Amazon Glacier

#203

Earlier quoted context omitted.

Sorry, what I meant by "stable" there is countries that have been relatively stable for a few hundred years and seem reasonably likely to continue that integrity for at least a century or so. Of course, predicting future stability is complete guesswork!

US. UK. Sweden. It is really hard to come up with countries where the government has been stable "for a couple of centuries".

Even the US had a close call with a fairly nasty civil war in that time frame, and in three days the 198th anniversary of the Burning of Washington happens: http://en.wikipedia.org/wiki/Burning_of_Washington "On August 24, 1814, after defeating the Americans at the Battle of Bladensburg, a British force led by Major General Robert Ross occupied Washington, D.C. and set fire to many public buildings. The facilities of the U.S. government, including the White House and U.S. Capitol, were largely destroyed."

Re: Amazon Glacier

#204
post #76

Interestingly they penalise you for short-term storage: Amazon Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you…

After reading tezza's explanation [1] of how they're probably using tape storage, this makes sense; Amazon wants the mechanical robot arm to spend the majority of its time writing to the tapes. If you're constantly tying it up with writes/deletes, you're taking time away from its primary mission: to archive your data. Charging you for early deletes discourages that practice.

[1] http://news.ycombinator.com/item?id=4411697

Re: Amazon Glacier

#205

Earlier quoted context omitted.

Sorry, what I meant by "stable" there is countries that have been relatively stable for a few hundred years and seem reasonably likely to continue that integrity for at least a century or so. Of course, predicting future stability is complete guesswork!

US. UK. Sweden. It is really hard to come up with countries where the government has been stable "for a couple of centuries".

Thailand.

Re: Amazon Glacier

#206

Earlier quoted context omitted.

I use a combination of Dropbox and S3. I have my MacBook and a Linux server linked to my Dropbox account. So changes in my documents are synced to my Linux. My Linux run three cron-jobs. One daily, one weekly and another monthly. The command is. s3cmd sync --delete-removed ~/Dropbox/documents/ s3://backup-daily/ There are buckets for weekly and monthly too. Note: the command is not exactly like that, check the man pa…

Wait, do I understand this correctly: You sync your dropbox to S3? Do you overwrite your previous daily when you do so?

Yes I do it, I have the weekly and monthly.

And yes, I might have daily-monday / daily/tuesday.... and so on, and it would work better, but this way works for me. There is a lot of room for improvement, and it is not hard to implement with cron-jobs and more buckets.

Re: Amazon Glacier

#207
post #28

Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month. What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics. Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.

"Paying $12 to store a gigabyte of data for 100 years" I'm not sure what kind of organisation I'd actually trust to store data for that length of time - a commercial organisation is probably going to be more effective at providing service but what commercial organisation would you trust to provide consistent service for 100 years? A Swiss bank perhaps? Governments of stable countries are obviously capable of this (cl…

A foundation or trust. For example, the Nobel Foundation has operated since 1900 without appearing in danger of closing. http://en.wikipedia.org/wiki/Nobel_Prize#Nobel_Foundation

You could have a foundation established to choose and pay an array of commercial organizations that do the archiving.

Re: Amazon Glacier

#208
post #79
post #47

Earlier quoted context omitted.

Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.

I'd guess that they ignore that problem and have baked the time it takes to get data from several tapes into the 3-4 hour estimate. If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time r…

That's true, but they're obviously using a multi-tier solution, with about a 90 day buffer before things go to tape:

  In addition, there is a pro-rated charge of $0.033 per gigabyte for items deleted prior to 90 days.
So it may just be feasible to organise writes so they end up together. But, yes, ultimately it is probably not worth it to do so.

Re: Amazon Glacier

#209

Earlier quoted context omitted.

If you're in a corporate environment, you'd likely have a network admin group that could throttle your connection to them. Or you'd be using a custom-designed front-end that would throttle it. Or if you're SOHO, you could set up QoS rules on your router to throttle it. Realistically though, this service might not be for you if fast and cheap retrieval of your data is important. The importance here is cheap storage, n…

Oh I totally understand that people are going to be willing to pay more after a disaster to get files back. But realistically, if there is a large enough volume of files to bother amazon then they're going to need day(s) to download. If they rate-limited by day then the price would only reach a couple years of storage. The hourly thing is only going to bite minor retrieval events, and it is going to bite them amazing…

My department (information security) was actually just discussing this service this morning at our morning meeting. We've been looking into backup services for our security monitoring appliance beyond our datacenter and DR site. These backups would need to persist for a year according to PCI/SOX compliance, and if we needed to show data to an auditor, we wouldn't need the entire log. In fact, we'd likely not even need 5% of it. Most likely, we'd only need to pull a day (maybe two) from the logs.

I can imagine our media services group talking about the same thing, how to keep master files of their product pictures/videos where they'd only need to grab a file or two here or there (if at all).

Amazon seems to be pushing this as a file dump where retrieval of the files is exceedingly rare. They don't want you to use it as a hard drive, they want you to use it as a magnetic tape drive.

Re: Amazon Glacier

#210
post #76

Interestingly they penalise you for short-term storage: Amazon Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you…

Likely they are losing money on the cost of initially writing data, then profiting from storage. With free early deletion they would lose money.
Post reply on HN