Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Glacier

aws.amazon.com

Post reply on HN
Print view View original

241–250 of 393 posts

Re: Amazon Glacier

#241

Post by flyt »

freehunter wrote: post #189
tommoor wrote: post #61

Earlier quoted context omitted.

agreed, it would be great to know how this is running from a hardware point of view - just out of personal interest :-)

Someone further up mentioned a very plausible (in my experience) answer. Magnetic tape, using hard drive arrays as RAM. The wait time in this situation would be the time needed to complete all the current tasks waiting to be written/read in the queue before your data is written from tape to hard drive so you can access it.

Amazon specifically said it does not use tapes.

Re: Amazon Glacier

#243

Post by flyt »

nemesisj wrote: post #7

I had a quick skim through the marketing stuff and the FAQs and didn't see anywhere that actually details what the backend of this is. I'd be curious if they're actually using tape, older machines, Backblaze pods, etc. I guess if it's the latter, the time to recover could be an artificial barrier to prevent people from getting cute.

It appears to use S3 as its basic backend. My guess is that S3 has been modified to have "zones" of data storage that can be allocated for Glacier. Once these zones have been filled with data (and of course that data is replicated to another region) the hard drives are spun down and essentially turned off.

This is why the cost of retrieval is so high: every time they need to pull data the drives need to be spun back up (including drives holding data for people other than you), accessed, pulled from, then spun back down and put to sleep. Doing this frequently will put more wear and tear on the components and cost Amazon money in power utilization.

As is Glacier should be extremely cheap for AWS to operate, regardless of the total amount of data stored with it. Beyond the initial cost of purchasing hard drives, installing, and configuring them the usual ongoing maintenance and power requirements go away.

Re: Amazon Glacier

#244

Post by varikin »

chmars wrote: post #232
leejoramo wrote: post #228

Earlier quoted context omitted.

CrashPlan defaults to using the account password as the encryption password. However, you can also secure the encryption with a password not associated with the account. Or even provide your own 448-bit key. If you do either of these options, CrashPlan support will not be able to help you This setup allows CrashPlan to easily help non-technical home users, while allowing technically savvy users to securely hang thems…

My question is not about the setup, that's OK. I am wondering why CrashPlan shows the encryption key in the clear and does not store it in the user or system keychain.

As others have noted, it is partially because Crashplan is a Java based app. Partially, it is also because Crashplan runs as System, not as the user. That way, I can have Crashplan backup my wife's user account and my user account.

Furthermore, you can use the encryption key + a custom password, or your own encryption key with a passphrase. In this case, it is encyrpted locally and the key is not sent to Crashplan[1].

[1] http://support.crashplan.com/doku.php/recipe/change_security...

Re: Amazon Glacier

#245

Post by antoko »

tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

I think you're making an incorrect assumption about which is the most plausible method for calculating the hourly retrieval rate.

The most obvious way to me would be to assume it is based on the actual amount of data transferred in an hour less the free allowance they give you. Which is actually what they say:

"we determine the hour during those days in which you retrieved the most amount of data for the month."

This also ties in with what the cost is to them, the amount of bandwidth you're using.

In your example you would need to be getting transfer rates of 3TB/hr. Given the nature of the service I don't think they are offering that amount of bandwidth to begin with. (I'm sure they get good transfer rates to other amazon cloud services but customers could be downloading that data to a home PC at which point they will not be getting anything even close to those transfer rates)

At that point a bigger issue might be how long it takes to get the data out rather than the cost.

At an overly generous download speed (residential cable) of 10GB/hr your 3TB archive would take over 12 days to download.

Re: Amazon Glacier

#246

Post by pnathan »

zach wrote: post #28

Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month. What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics. Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.

This seems like a very interesting business idea. It'd require some level of initial operating capital and a relatively competent server farm team, but I don't think it'd have to be fancy.

"Long Data, LLC... We secure your data for the long-term".

Re: Amazon Glacier

#247

Post by Loic »

tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

Good catch, in fact it totally fits with the description of the service as a store and forget for compliance and access only a small subset in the case of retrieval requests — for example when storing customer records.

I also must say that the way you calculate the retrieval fee is really looking like black magic at first sight. I hope they will add a simple calculator to evaluate some scenario and provide the expected bandwidth available from Glacier to an EC2 instance.

Re: Amazon Glacier

#248

Post by ratzkewatzke »

tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

First, the reason the "peak hourly retrieval rate" of "1 gigabyte per hour" is there in the article is to answer this question. At a relative allowance of 5.12 GB/day and 1 GB/hour transfer rate, that gives you a "peak hourly retrieval" of .79 GB (at 5.12/24, your first .21 is free), and so we multiply:

.79 * 720 * .01

Giving me a little less than $6.

Now, do you think Amazon is likely to think they can get away with selling a service that charges you $22k for a 3TB retrieval?

Second, you have ranged GETs and tape headers; use them to avoid transferring all of your data out of the system at once. [Edit: looks like ranged GETs are on job data, not on archival retrieval itself. My bad.]

Re: Amazon Glacier

#249

Post by metalruler »

gjm11 wrote: post #175
Jabbles wrote: post #137

Earlier quoted context omitted.

per object . So although the chance of losing any particular object is tiny, the chance of you losing something is proportional† to the number of objects. Still extremely small. †roughly proportional if you have << 1e11 objects

Yes. Though I bet the real lossage probabilities are dominated by failure events that take out a substantial fraction of all the objects there are, and that happen a lot more often than once per 10^11 years.

Agreed. More likely a catastrophic and significant loss for a small number of customers rather than a fraction of a percentage of loss for a large number.

Similar deal for hard drive bit error rates, where the quoted average BER may not necessarily accurately represent what can happen in the real world. For example, an unrecoverable read error loses 4096 bits (512 byte sectors) or 32768 bits (4k sectors) all at once, rather than individual bits randomly flipped over a long period.

Re: Amazon Glacier

#250

Post by thebigshane »

arethuza wrote: post #127
zach wrote: post #121

Earlier quoted context omitted.

Yeah, I'm kind of wondering the same thing. It's certainly the kind of timeframe that changes your perspective. Maybe a tiny bit of Danny Hillis rubbed off on me from working at Applied Minds (man, I sure hope so!) Because as we answer issues of cost and availability, a logical thing to wonder is "how long can I really depend on it though?" As quickly as cloud services (where "lifetimes" are measured at six years) ha…

Probably what would be required is an array of arrays of separate storage providers and services providing "RAID" on top of these storage providers - and you won't want to trust any of these you'll want a few of them... (hence the array of arrays).

Wow, does any service offer something like that? RAID6 would be pretty interesting across s3, google drive, (hrmm, what else? skydrive? rsync.net?)
Post reply on HN