Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Glacier

aws.amazon.com

Post reply on HN
Print view View original

371–380 of 393 posts

Re: Amazon Glacier

#371

Post by wamatt »

vl wrote: post #303
res0nat0r wrote: post #287

Earlier quoted context omitted.

Based on the ZDNet article linked elsewhere on the comments, this system does not use any tape at all. It is all commodity hardware and hard drives, pretty much in line with the design of the rest of the services from AWS.

But why there is retrieval delay then?

I suspect it may actually be related to the energy cost of higher data rates on the device itself, rather than the network costs.

IOW, it would require more energy to spin the disks faster, and burst a higher peak rate.

Re: Amazon Glacier

#372

Post by dredmorbius »

philwelch wrote: post #335
dredmorbius wrote: post #313

Earlier quoted context omitted.

The Thai king is the longest-reigning current head of state, ascending the throne on 9 June 1946. Elizabeth II of England is 2nd, 6 February 1952. The oldest country (not government) is likely Vietnam (2897 BCE). Other contenders: Japan (660 BCE), China (221 BCE), Ethiopia (~800 BCE), or Iran (678 BCE). Few of today's modern states pre-date the 19th Century, many antedate World War II or the great de-colonialisation…

Isn't Egypt another contender for the oldest country?

That was my thought as well. However it spent a great deal of time under foreign rule: under the Greeks and Romans, the Turkish / Ottoman empire, and later under British occupation. And, I just discovered what boxer Muhammad Ali's referent was.

http://en.wikipedia.org/wiki/Egypt_history

Re: Amazon Glacier

#373

Post by sintaks »

ghshephard wrote: post #355
sintaks wrote: post #339

Earlier quoted context omitted.

Former S3 employee here. I was on my way out of the company just after the storage engineering work was completed, before they had finalized the API design and pricing structure, so my POV may be slightly out of date, but I will say this: they're out to replace tape. No more custom build-outs with temperature-controlled rooms of tapes and robots and costly tech support. If you're not an Iron Mountain customer, this p…

The math doesn't come close to replacing tape - basically once you go north of 100 terabytes (just two containers - at my prior company we had 140 containers in rotation with iron mountain) Glacier doesn't make financial or logistical sense. Far cheaper and faster to send your LTO-5 drives via driver.

It may not make sense today. Amazon is notorious for betting on the far future. They're also raising the bar on what archival data storage services could offer. When you ship your bits to Amazon, they're in 3+ DCs, and available programmatically.

Separate from the play for replacing tape, there's also the ecosystem strategy. When you run large portions of your business using Amazon's services, you tend to generate a lot of data that ends up needing to be purged, else your storage bill goes through the roof. S3's Lifecycle Policy feature is a hint at the direction they want you to go - keep your data, just put it somewhere cheaper.

This could also be the case where they think they're going after tape, but end up filling some other, unforeseen need. S3 itself was originally designed as an internal service for saving and retrieving software configuration files. They thought it would be a wonder if they managed to store more than a few GB of data. Now look at it. They're handling 500k+ requests per second, and you can, at your leisure, upload a 5 TB object, no prob.

But maybe you're right. The thing could fail. Too expensive. After all, 512k ought to be enough for anybody.

Re: Amazon Glacier

#374

Post by rektide »

tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

3TB is a huge archive. I'm also not sure about your maths, billable peak hourly chiefly. [ed: dot multiplier for formatting]

Let's run 100GB, X. Allowance limit: 100GB . 5% is 5GB/mo, or per day, 100GB/(2030), 0.166GB/day; X/600.

Hourly rate necessary for a sustained 24 hour cycle of 100GB is: 100GB/24hr, or 4.166GB/hr, X/24. Peak hourly, this.

To determine the amount of data you get for free, we look at the amount of data retrieved during your peak day and calculate the percentage of data that was retrieved during your peak hour. We then multiply that percentage by your free daily allowance.*

To begin all that's stated here is, break your data-retrieval out over a day. Their example:

you retrieved 24 gigabytes during the day and 1 gigabyte at the peak hour, which is 1/24 or ~4% of your data during your peak hour.

We're doing 4.166GB in the peak hour/100GB in the peak hour, or ~4%.

X/24 / X = 1/24 = ~4.1666666% if you don't fuck your meteringly up.

We multiply 4% by your daily free allowance, which is 20.5 gigabytes each day. This equals 0.82 gigabytes [ed: free allowance hourly]. We then subtract your free allowance from your peak usage to determine your billable peak.

Free allowance hourly rate: 4.16666% . 0.166 = 0.006666, or (X/600/24), X/15000. A is at (12 . 1024)/15000, or indeed 0.8192 free, to verify.

billable peak hourly is then: hourly peak rate - free rate, 4.1666 - 0.00666 = 4.160, or (X/24) - (X/600/24) or (X-(X/600))/24 or (599X/600)/24 or simply, billable peak hourly will always be for sufficiently non-incompetent implementations: ~0.0415972222X. Always.

Let's check: 100GB . 0.041597 = 4.15970. Cannot compare to amazon, because their hourly rate is calculating a 24GB of 12TB archive download, but, 1-0.8192 still checks out. It would be 511.14666666 if their entire set, or (12 . 1024)/24 - 0.8192, 511.1808GB/hr peak hourly (nice pipes kids).

Retrieval fee is then, 0.041597X . 720 . tier pricing, and tier pricing I really do not understand the origin of at all but all examples seem to be 0.01. So, $29.95/100GB. For 12TB, say hello to $3680.25599 transfer fee. 3TB is $920.064.

720 . (599X/600) /24 /100, so for the transfer of your entire set X GB of data, evenly done across the day, you will be charged: (599X/600).(3/10)$,

0.2995$/GB to pull data out in a day.

Re: Amazon Glacier

#375

Post by tomkarlo »

sintaks wrote: post #342
tomkarlo wrote: post #328

Earlier quoted context omitted.

I'm not talking load so much as network traffic. Companies like Amazon and Google have huge peak hour outbound traffic during US waking hours, and then a huge dip during off hours. If they can push more of the traffic into those off hours they can make the marginal cost of the bandwidth basically zero. So if you make a request at peak hours (say 12 noon ET), they just make you wait until 11 PM ET to start downloading…

Amazon has ridiculous internal bandwidth. The costly bit is external. The time delay is largely internal buffer time - they need to pull your data out of Glacier (a somewhat slow process) and move it to staging storage. Their staging servers can handle the load, even at peak. GETs are super easy for them, and given that you'll be pulling down a multi-TB file via the Internet, your request will likely span multiple da…

I was referring to the external bandwidth. Even if pulling down a request takes hours, forcing them to start off peak will significantly shift the impact of the incremental demand. I'm guessing that most download requests won't be for your entire archive - someone might have multiple months of rolling backups on Glacier, but it's unlikely they'd ever retrieve more than one set at a time. And in some cases, you might only be retrieving the data for a single use or drive at a time, so it might be 1TB or less. A corporation with fiber could download that in a matter of hours or less.

Re: Amazon Glacier

#376

Post by philwelch »

dredmorbius wrote: post #372
philwelch wrote: post #335

Earlier quoted context omitted.

Isn't Egypt another contender for the oldest country?

That was my thought as well. However it spent a great deal of time under foreign rule: under the Greeks and Romans, the Turkish / Ottoman empire, and later under British occupation. And, I just discovered what boxer Muhammad Ali's referent was. http://en.wikipedia.org/wiki/Egypt_history

So have Vietnam, China, and Iran.

Re: Amazon Glacier

#377

Post by rektide »

sintaks wrote: post #339
tc wrote: post #222

Beware that retrieval fee! The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1]. It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour? This…

Former S3 employee here. I was on my way out of the company just after the storage engineering work was completed, before they had finalized the API design and pricing structure, so my POV may be slightly out of date, but I will say this: they're out to replace tape. No more custom build-outs with temperature-controlled rooms of tapes and robots and costly tech support. If you're not an Iron Mountain customer, this p…

I wasn't holding my breath, but I was thinking there's a possibility they were using short-stroking to speed up most of their systems hard drives by making a quarantined barely touched Glacier zone in the inside of their drives: https://plus.google.com/113218107235105855584/posts/Lck3MX2G...

My backup wouldn't it be cool if is, unlike the above reasonableness, a joke: imagining 108 USB hard drives chained to a poor PandaBoard ES, running a fistful at a time: https://plus.google.com/113218107235105855584/posts/BJUJUVBh...

The Marvell ARM chipsets at least have SATA built in, but I'm not sure if you can keep chaining out port expanders ad-infinitum the same way you can USB. ;)

Thanks so much for your words. I'm nearly certain the custom logic boards you mention are done with far more vision, panache, and big-scale bottom line foresight than these ideas, even some CPLD multiplexers hotswapping drives would be a sizable power win over SATA port expanders and USB hubs. Check out the port expanders on OpenCompute Vault 1.0, and their burly aluminium heat sinks: https://www.facebook.com/photo.php?fbid=10151285070574606&#3...

Re: Amazon Glacier

#378

Post by dredmorbius »

philwelch wrote: post #376
dredmorbius wrote: post #372

Earlier quoted context omitted.

That was my thought as well. However it spent a great deal of time under foreign rule: under the Greeks and Romans, the Turkish / Ottoman empire, and later under British occupation. And, I just discovered what boxer Muhammad Ali's referent was. http://en.wikipedia.org/wiki/Egypt_history

So have Vietnam, China, and Iran.

Japan's older than China. But yes. As I mentioned in my post above.

Re: Amazon Glacier

#379

Post by rektide »

ghshephard wrote: post #36

I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage. With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,v…

A possible alternative to Glacier+product X (you mention Arq) could perhaps be Git-Annex, recently mentioned as it's Kickstarting a friendly UI component: http://news.ycombinator.com/item?id=4385886 http://www.kickstarter.com/projects/joeyh/git-annex-assistan...

Already has Amazon support: http://git-annex.branchable.com/tips/using_Amazon_S3/

Very friendly with very large files!

Re: Amazon Glacier

#380

Post by rektide »

cayblood wrote: post #54
martey wrote: post #16

I think one of the most interesting parts of this is how they plan to ensure that people do not use it for transient backup: https://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_de... Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US…

So I guess that means it would still work well for a scheme like time machine uses, where incremental changes are added but deletions are simply made note of. At least I think that's how it works.

And yet they support a max of 100 vaults per account, so some roll behind recompaction of incrementals is still necessary.
Post reply on HN