51–55 of 55 posts
A little off topic, but it seems really strange to me that Amazon is not transparent on the technology. Because of the high charge for fast reads, I tend to believe that the underlying storage is some form of media that gets mounted, perhaps like Facebook's bluray archival system.
Earlier quoted context omitted.
Emphasizing the above, the study a few years ago of disk drives at big HPC centers found that they don't follow a bathtub failure model. Rather, there are very few infant failures, and wear is noticeable starting in the range of 1-2 years in service.
The google results I saw were more like an exponential curve, i.e. a memory-less distribution - which would mean old drives are just as reliable as new ones.
What I'd like to see is a storage product that aggregates the unused storage space on EC2 instances. * By default instance storage isn't attached, so there's probably a lot of completely available capacity. * Even if attached, it's rare that the full capacity would be used, so thin provisioning would leave some space available. * Some host machines won't be fully allocated. I imagine taking this pool of capacity and…
They see a computer with an attached disk of, say, 1 TB. They don't see the filesystem at all. They don't know if a sequence of zeros is unused space, or are literally zeros in data. There is no way that Amazon can use slack space because they don't know what is slack space.
This incomplete view of how disks (both instance and EBS) are used is also evident by what metrics CloudFront can track: CPU usage. Raw I/O stats for disk and network. To get richer metrics, you have to install scripts/agents running inside your EC2 instance which than beacon the info to CloudFront via the API.
In short, Amazon's current setup means they cannot use the slack space inside of attached block devices, even if they wanted to.
another question: what is the difference in power consumption between a HDD that's powered up but idle, and one that's being used to read from or write to? As an aside, erasure codes allow you to reduce power consumption as the redundant fragments are only necessary for safety, not for regular retrieval. You don't need glacier to benefit from that. (But glacier might be an optimization of that strategy)
What I'd like to see is a storage product that aggregates the unused storage space on EC2 instances. * By default instance storage isn't attached, so there's probably a lot of completely available capacity. * Even if attached, it's rare that the full capacity would be used, so thin provisioning would leave some space available. * Some host machines won't be fully allocated. I imagine taking this pool of capacity and…
Maybe for unattached instance storage this is possible (thought not a good business decision as others have noted). But you are forgetting how Amazon sees those attached instance disks: As raw block devices from the hypervisor. They see a computer with an attached disk of, say, 1 TB. They don't see the filesystem at all. They don't know if a sequence of zeros is unused space, or are literally zeros in data. There is…
As to whether EC2 does this I don't know - based on your observations they may not. I think it's an interesting way to build a storage product like Glacier, even if Amazon may have chosen to do something different.