Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Glacier redux

storagemojo.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Glacier redux

#11

Post by greatzebu »

duskwuff wrote: post #9
greatzebu wrote: post #7

If you have a lot of heavily-read data distributed across machines, you're probably constrained by available spindles rather than available storage space. So co-locating data that will almost never be read with heavily-read data is effectively free. I would guess that Amazon stores Glacier data alongside S3 data, and the lower price reflects the fact that the limiting factor in their storage system is IO rather than…

That doesn't offer any clear explanation of why they would charge extra for early deletion of that data, though.

One possible explanation: the cost of initial IO to write your data is amortized over that data's lifetime. The early-delete penalty ensures that Amazon always makes enough on your data to justify the cost of writing it into their system in the first place.

Re: Glacier redux

#12

Post by Cymen »

michaelt wrote: post #10

But NONE of the Hacker News commenters addressed Sony and Panasonic’s continued investment in high-density optical disc technology. [...] There has to be a business reason for the continued investment, i.e. customers prepared to buy a lot of product in the future and buying a lot right now. If Amazon was the only customer for high-density optical storage, they'd be crazy to invest in developing it, because Amazon wil…

Well a couple of other organizations that have lots of data and don't want to share their practices come to mind: NSA, FBI, etc.

Re: Glacier redux

#13

Post by enricotal »

The Amazon Glacier Secret is hidden in plan sight. I think is just a virtual product. Probably unused S3 capacity, at a much lower price, not a different technology.

Re: Glacier redux

#14

Post by eli »

michaelt wrote: post #10

But NONE of the Hacker News commenters addressed Sony and Panasonic’s continued investment in high-density optical disc technology. [...] There has to be a business reason for the continued investment, i.e. customers prepared to buy a lot of product in the future and buying a lot right now. If Amazon was the only customer for high-density optical storage, they'd be crazy to invest in developing it, because Amazon wil…

Is it really that unusual to have two vendors competing for a single very large customer?

Re: Glacier redux

#15

Post by watson »

I've always thought that the deletion fee for <3 month old data was a way to limit "misuse of the service" - call it a detergent if you will - more than actually a way to recuperating costs. Because I guess the costs will be there no matter when you delete data

Re: Glacier redux

#16

Post by dmourati »

Other than burying the cite (2 links to his own blog) to why "cost of provisioning a single watt of power is more expensive than 10 years of power consumption," the author's canonical source is eventually a 404:

http://static.googleusercontent.com/media/labs.google.com/en...

The data is also > 7 years old.

EDIT: working link:

http://static.googleusercontent.com/media/research.google.co...

Re: Glacier redux

#17

Post by watson »

xhrpost wrote: post #5

The deletion charge is interesting. Does deletion guarantee a scrubbing of the data as soon as possible? If so, I could see that as justification for the fee since some sort of significant work is involved in retrieving data, implied by the hours of wait time. If not though, the author has a good point in that Amazon needs to make at least .03/GB in order to be profitable.

I would be surprised. Anyway, why would they pay for scrubbing if the data is less than 3 months old and do it for free if not? If they do scrub, I don't think those are related

Re: Glacier redux

#18

Post by timfrietas » 

    Glacier is significantly cheaper than S3
Yes, as long as you put something in and almost never take it out.

    They charge for deletions in the first 3 months
What if this is just disincentive to pull content out and treat Glacier like S3?

    Power is not the driving cost for Internet scale infrastructure
It is not the only cost, but it is still one of the largest factors, no?

    Sony and Panasonic continue to invest in a product that has no visible commercial uptake
That means nothing in and of itself. The only person who will win such a market needs to be one of the first there, innovator's dilemma, etc.

    Facebook believes optical is a reasonable solution to their archive needs
Do they? I saw one mention in the author's previous post of James Hamilton commenting on a Facebook cold storage system using Blu-ray but is unclear to me if it is in production.

Assuming it is true though, it is likely an apples-to-oranges comparison. Glacier provides archival restoration for presumably largely enterprise-level customers. Facebook backs up data from users, and I'd presume this is from deactivated accounts, etc., and unlikely to need urgent restoration.

Re: Glacier redux

#19

Post by reitzensteinm »

The author handwaves away the power savings of powering down disks by talking about the capital cost of providing power.

"Unless the prices of copper, PDUs and diesel-generators have started following Moore’s Law, this is probably more true today than in 2007."

But this is fallaciously assuming that Glacier servers would need to be like EC2 or S3 servers that are switched on for x random hours per day. This isn't the case at all.

Diesel generators, for instance - what's the use? If you're replicating the data around the world, it doesn't matter if your Glacier servers in one location are powered down for days.

Cooling, power - just switch off every single Glacier server at each location during the peak hours of the day. No additional capital costs, because you're never adding to peak power usage.

Distribution of reads - Profile customers based on how frequently they're reading from the data store. I bet 50% of customers never read a single byte back. Colocate these customers on the same servers, and you only have to power up the server once a month to check if the data is still there.

Bonus - Profile data access patterns with simple heuristics to determine what's likely to be read back in, and temporarily store that data in S3. Imagine a company that archives everything to Glacier daily, but restores day, week and month old backups regularly. Keep all data less than a month old in S3, and the rest on the indefinitely powered down servers talked about above.

I actually quite liked the author's case for BDXL, but it seems he's doing a straw man on other possible solutions. Well implemented BDXL more cost efficient than a naively implemented disk strategy? Not exactly news.

Post reply on HN