Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Glacier redux

storagemojo.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Glacier redux

#41

Post by snowwrestler »

If the goal is to manage capital costs, I don't see how funding the development of an entirely new data storage hardware ecosystem is a reasonable answer. You control capital costs with commodity hardware, not cutting edge.

Yet that is what the author thinks Amazon is doing with BDXL.

So who's funding high-capacity optical storage? Hmm, can we think of a customer who ingests huge amounts of data, wants to keep it for a long time, and has no fear if funding cutting-edge product development? Yes: defense departments.

Re: Glacier redux

#42

Post by keypusher »

raverbashing wrote: post #39
reitzensteinm wrote: post #35

Earlier quoted context omitted.

It's not about the price they're paying for power (peak hours from the power company's perspective), it's about the peak usage of their data center. As the article (correctly) says, the capital costs are dependent on your peak usage. If you have 1000 servers using 400kW at peak, you need sufficient air conditioning to extract 400kW worth of heat, and backup generation capable of producing 400kW. It doesn't matter if…

The real question then is: is there really a peak hour for datacenter usage/consumption? What's the consumption difference between peak and regular hours (and low-demand hours) How much power does Amazon use during mornings compared to Netflix watching peak time?

Yes, I'm sure there is. Look at Google clicks by hour or similar metrics, there is a significant curve during the day, peak can be 4-5x the low.

Re: Glacier redux

#43

Post by atdt »

Is data really so heterogeneous? How many files in your home directory are generic, with bit-exact duplicates existing in the home directories of many other users? And the remaining files, which are uniquely yours -- what percent of each of them consists of generic data, like file headers?

Re: Glacier redux

#44

Post by GregorStocks »

justinsb wrote: post #36

What I'd like to see is a storage product that aggregates the unused storage space on EC2 instances. * By default instance storage isn't attached, so there's probably a lot of completely available capacity. * Even if attached, it's rare that the full capacity would be used, so thin provisioning would leave some space available. * Some host machines won't be fully allocated. I imagine taking this pool of capacity and…

I don't see why Amazon would want to buy enough disks to support 100% of what they promise and then sell the unused capacity at below cost when they could just buy fewer disks instead. Either way they'd be in trouble if their EC2 customers suddenly started wanting to use all the space they were promised.

Re: Glacier redux

#45

Post by richardw »

If power provisioning and space are an issue it seems unlikely to me that they're sitting with many drives all hooked up for immediate power-up. It would make more sense to separate the storage of media and the power connection so you get the most from each power connection. Therefore: removable media, whether those are hard drives or optical.

There's a data retrieval period of multiple hours [1]. That doesn't sound like they're just powering up a drive. It sounds like they're moving something around or doing some kind of linear read (as opposed to random-access). I'd bet on "a retrieval job fetches this stack of read-only media and connects it to the powered device, which loads it onto hard drives for quick download".

[1] "Retrieval jobs typically complete within 3-5 hours" - http://aws.amazon.com/glacier/faqs/

Re: Glacier redux

#46

Post by justinsb »

GregorStocks wrote: post #44
justinsb wrote: post #36

What I'd like to see is a storage product that aggregates the unused storage space on EC2 instances. * By default instance storage isn't attached, so there's probably a lot of completely available capacity. * Even if attached, it's rare that the full capacity would be used, so thin provisioning would leave some space available. * Some host machines won't be fully allocated. I imagine taking this pool of capacity and…

I don't see why Amazon would want to buy enough disks to support 100% of what they promise and then sell the unused capacity at below cost when they could just buy fewer disks instead. Either way they'd be in trouble if their EC2 customers suddenly started wanting to use all the space they were promised.

It's a good point: a cloud provider could choose to try to under-provision disks instead. The problem though is that (some) disks are local to the machine; it's not easy to move physical hard disks around when the calculations are wrong (and if you have to get it right on a per-host basis, it's more likely to go wrong). It is however easy to move chunks of data around, particularly if you use something like erasure coding to give you a huge amount of flexibility.

In short, your way is a good alternative, but my guess is that buying the full capacity and selling the surplus is probably roughly cost-equivalent, and considerably less likely to end up with you not being able to sell the full capacity of any given host.

Re: Glacier redux

#47

Post by jpalomaki »

raverbashing wrote: post #31
reitzensteinm wrote: post #19

The author handwaves away the power savings of powering down disks by talking about the capital cost of providing power. "Unless the prices of copper, PDUs and diesel-generators have started following Moore’s Law, this is probably more true today than in 2007." But this is fallaciously assuming that Glacier servers would need to be like EC2 or S3 servers that are switched on for x random hours per day. This isn't the…

Makes me wonder something Database machines are usually never shut down. And when they are, it's usually a manual process (unless we're talking about a power failure or something) "Diesel generators, for instance - what's the use?" Backup power (not only for Glacier). Reducing power consumption in peak hours is a good strategy, however, I'm not sure how worried Amazon is about this (and the difference in price betwee…

I think the requirements for power are much different in case of glacier that with some more generic computing. With a service like glacier, running on hard disks, you could probably completely turn off large parts of your infrastructure. With 5 hour read latency there should be plenty of opportunities to optimize the reads so that only a small part of the infra needs to be powered on at any time.

Also you might be able to skip the backup power part completely. The data is anyways distributed to few locations. The probability of many of them losing to power at the same time for several hours is likely to be very small.

Maybe there's also large differences in how the customers use the system. I would assume some customers are more likely to retrieve data than others. Once you start learning these patterns you could further optimize the storage.

The pricing of Glacier is not necessarily driven by technical reasons. Amazon already has S3 and they need to differentiate the products.

Re: Glacier redux

#48

Post by lmm »

hga wrote: post #6
WestCoastJustin wrote: post #4

> Old hard drives that are no longer economical for more intensive service, supported by disk-handling robotics. I am with the author in that I do not think they are using older disks. From personal experience (working with petabytes of disk based storage), I would put money on the fact that AWS is not using stock off the shelf older disks and then powering them on/off as a method of storage. The last thing you want…

Emphasizing the above, the study a few years ago of disk drives at big HPC centers found that they don't follow a bathtub failure model. Rather, there are very few infant failures, and wear is noticeable starting in the range of 1-2 years in service.

The google results I saw were more like an exponential curve, i.e. a memory-less distribution - which would mean old drives are just as reliable as new ones.

Re: Glacier redux

#49

Post by lmm »

atdt wrote: post #43

Is data really so heterogeneous? How many files in your home directory are generic, with bit-exact duplicates existing in the home directories of many other users? And the remaining files, which are uniquely yours -- what percent of each of them consists of generic data, like file headers?

I have 100gb of gaming screencaptures. That data exists nowhere else (some people may have copies of the final cuts on youtube, but that's tiny in comparison); if you could store it as input data + game code you could compress it by a lot, but I highly doubt Amazon's that far ahead of mainline video encoding technology.

Other than that, I think the big culprit will be photos; everyone's family photos are different (the JPEG header is a tiny proportion of a modern 5MB photo) and that's one of the most popular things for people to back up on these kind of services.

Plenty of data is generic in the way you say, but plenty of it isn't. So I don't think there's any free lunch here.

Re: Glacier redux

#50

Post by toolslive »

another question: what is the difference in power consumption between a HDD that's powered up but idle, and one that's being used to read from or write to?

As an aside, erasure codes allow you to reduce power consumption as the redundant fragments are only necessary for safety, not for regular retrieval. You don't need glacier to benefit from that. (But glacier might be an optimization of that strategy)

Post reply on HN