Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Glacier

aws.amazon.com

Post reply on HN
Print view View original

361–370 of 393 posts

Re: Amazon Glacier

#361

Post by chmars »

leejoramo wrote: post #223
sreitshamer wrote: post #160

Earlier quoted context omitted.

I'm looking into supporting Glacier in Arq. It sure is cheap -- $10/month for a terabyte.

Arq is great backup software. I would view inclusion of Glacier support as a worthy paid upgrade. Arq is the pinnacle of my rather large backup pyramid which also includes: Dropbox, Superduper, Crashplan, rsync, SVN/GIT and more.

In an earlier version of Arq, there was no possibility to see which data was actually selected for backup (and which not). Has Arq become more user-friendly?

Re: Amazon Glacier

#362

Post by mslot »

tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

Seems this is the right formula for estimating monthly cost:

0.01S+1.80R.max(0, 1-0.0017S/D)

S is number of GB stored.

R is the biggest retrieval of the month. Parallel retrievals are summed, even if overlap is only partial.

D is the amount of data retrieved on the peak day (≥R)

e.g., for 10TB storage, max. 50GB per retrieval, and max. 200GB retrieval per day: $2188.20 / year

http://www.wolframalpha.com/input/?i=0.01*10000%2B1.80*50*ma...

Re: Amazon Glacier

#363

Post by mslot »

mslot wrote: post #291

I think this is more or less the formula for calculating monthly costs (corrections welcome): 0.01S+max(0,7.20*(R-0.0017S)/4) S is number of GB stored R is biggest retrieval in the month 4 is the average number of hours a retrieval For an example with 10TB storage (replace 10000 to change): http://fooplot.com/plot/4pu7u2gpox x is biggest retrieval in GB, y is $/month

Correction: See my other post http://news.ycombinator.com/item?id=4416684

Re: Amazon Glacier

#364

Post by stream »

sreitshamer wrote: post #160
ghshephard wrote: post #36

I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage. With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,v…

I'm looking into supporting Glacier in Arq. It sure is cheap -- $10/month for a terabyte.

Arq, please add support for Glacier. - Happy customer.

Re: Amazon Glacier

#365

Post by digeridoo »

TylerE wrote: post #292
digeridoo wrote: post #283

Earlier quoted context omitted.

10Gbps EC2 instances start at $0.742/hour. Welcome to the cloud. ;-) I assume the cost is in retrieval though and counted per the Job Creation API, regardless of whether and how quickly you download the data. but you're right that the 3TB/hour use-case is very hypothetical. Internet archival is just not suitable for those kind of volumes. I think the point OP was making that mistakes like using archives that are too…

Well, yes and no. If you actually USE 10gpbs your data transfer bill is going to be around $167k per month (That's for transferring 3.34PB). Actually, a bit higher than that since I calculated all based on the cheapest tier EC2 will quote on the web, 5 cents per gigabyte. For a one time 3TB download to an EC2 instance, priced at the first pricing tier of $0.12/gigabyte, that transfer will cost $360, and take around 4…

Glacier to EC2 traffic is free if your instance is in the same region as the Glacier endpoint and $0.01/GB otherwise.

Re: Amazon Glacier

#366

Post by sreitshamer »

rglover wrote: post #219

I'm currently using an app called Arq that backs everything up to S3. If I had to guess, I'd say there's about 50-60 gigs or more on there. Last months bill was something like .60 cents. How does glacier compare or contrast to this setup (the app does something similar with the archive concept)?

I'd also like to know how the bill was so low (I'm the developer behind Arq). Is it perhaps because it's the first month's bill and you haven't had the 60GB on S3 for very long (not a full month)?

Re: Amazon Glacier

#367

Post by hga »

unreal37 wrote: post #307
ensignavenger wrote: post #296

Earlier quoted context omitted.

Online backup for my photos and other data was my initial thought, but I'm afraid it would cost too much to do a restore- if I store 3 TB of photos/documents/etc for 2 years, then have a house fire (local backup destroyed), I want to be able to restore my data to my new computer as quickly as my Internet connection will let me, and I don't want to be stuck with a huge bill for retrieval on top of all the other expens…

re: "I want to be able to restore my data to my new computer as quickly as my Internet connection will let me" Really? Why? If you have say 10 years of home pictures/movies, and you know they are 100% safe in Amazon Glacier, why do you need them all on your new computer as fast as possible? I don't understand why its such a rush. If it's a rush, you pay the fee. If you can afford to wait a month or two or three to ge…

Agreed. Let me add that a lot of us are living under severe data cap or overage regimes, for me and my parents it's 2 AT&T plain DSL lines, each with 150 GB/month free, go over that 2-3 times and you start paying $10/50 GB/month on a line.

So uploading as well as downloading would have to be throttled. But this sounds like a superb way to store all those pictures and movies of the grandchildren, especially for those who don't have a son with a LTO drive ^_^. All the other alternatives are lousy or a lot more expensive.

Re: Amazon Glacier

#368

Post by hga »

sintaks wrote: post #339
tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

Former S3 employee here. I was on my way out of the company just after the storage engineering work was completed, before they had finalized the API design and pricing structure, so my POV may be slightly out of date, but I will say this: they're out to replace tape. No more custom build-outs with temperature-controlled rooms of tapes and robots and costly tech support. If you're not an Iron Mountain customer, this p…

It wasn't built to back up your family photos and music collection.

But at its price points, with most US families living under pretty nasty data cap or overage regimes, it sounds superb, with of course the appropriate front ends.

There's no good (reliable), easy and cheap way to store digital movies, e.g. DVD recordable media is small by today's standards and it's much worse than CD-Rs for data retention (haven't been following Blu-ray recordable media, I must confess, I bought an LTO drive instead, but I'm of course unusual). And the last time I checked very few people made a point of buying the most reliable media of any of these formats.

In case of disk failure, fire, tornado (http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/ ... and rsync.net helped save the day), for this use case you don't care about quick recovery so much as knowing your data is safe (hopefully AWS has been careful enough about common mode failures) and knowing you can eventually get it all back. Plus a clever front end will allow for some prioritizing.

Important rule learned from Clayton Christensen's study of disruptive innovations (where the hardest data comes from the history of disk drives...) is that you, or rather AWS here, can't predict how your stuff will be used. So if they're pricing it according to their costs as you imply they're doing the right thing. Me, I've got a few thousand Taiyo Yuden CD-Rs who's data is probably going to find a second home on Glacier.

ADDED: Normal CDs can rot, getting them replaced after a disaster is a colossal pain even if your insurance company is the best in the US (USAA ... and I'm speaking from experience, with a 400+ line item claim that could have been 10 times as bad since most of my media losses were to limited water problems), so this is also a good solution to backing up them. Will have to think about DVDs....

Re: Amazon Glacier

#369

Post by duskwuff »

arethuza wrote: post #132
ghshephard wrote: post #125

Earlier quoted context omitted.

And, since you mentioned Danny Hills, it's also worth mentioning that (ironically?) Jeff Bezos is one of the principle supporters of http://longnow.org/clock/ .

And that makes me think of Anathem and the potential issues around long-term data storage that is capable of surviving through falls of civilisations and/or sacks of storage areas.

That's no coincidence -- Anathem was inspired by the Long Now Foundation and their 10,000 Year Clock project.

Re: Amazon Glacier

#370

Post by jlgreco »

DeepDuh wrote: post #126
Dylan16807 wrote: post #119

Earlier quoted context omitted.

The dropbox client itself keeps the old versions of files around for a few days. It doesn't matter if the backend gets confused.

Yes, I'm aware of that. This helps, up to the point where you don't notice that something's gone for 'a few days'. Let's say you have your student project's folder shared with two other colleagues. You take a few days off, use your PC for casual browsing, meanwhile your colleagues are working on the project. At the end, one of them (who doesn't quite understand how dropbox works) deletes the files while the other is…

If dropbox works well for you for source control, that is great. But frankly, if you get to the point where you have to start writing hacks to keep it working, it is probably time to move onto something designed for the task.
Post reply on HN