HGST gets closer to shipping 10TB HDD • Hacker News Board

Re: HGST gets closer to shipping 10TB HDD

#11

Post by fiatmoney » Fri, Mar 13, 2015, 2:22 AM UTC

the8472 wrote: post #5

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

Is there some fundamental obstacle that prevents raid rebuild times from being within a small, constant factor of linear scan time of a single disk? Naively I would think that it would simply have to stream large chunks of each drive in parallel, apply the parity coding and write out the stream of calculated blocks to the new disk.

Oh sure, it's a big linear streaming read/write - the issue is that if you're taking a 8, 12, 15, etc. hour window to rebuild, there's a fairly high chance of something else going wrong. Particularly because you have to worry that, eg, your flaky power source has been shortening drive lifetimes, that's why you saw the failure within the expected lifetime of the drive, and that's why the another one is going to die in hour 14 of your rebuild.

Of course, that's in the context of an actual RAID array (or ZFS pool, etc.) - if you've got a replication strategy that spreads multiple entire copies over multiple systems, you're less vulnerable and these kinds of disks are a good choice.

Re: HGST gets closer to shipping 10TB HDD

#12

Post by the8472 » Fri, Mar 13, 2015, 2:32 AM UTC

fiatmoney wrote: post #11

the8472 wrote: post #5

Earlier quoted context omitted.

Is there some fundamental obstacle that prevents raid rebuild times from being within a small, constant factor of linear scan time of a single disk? Naively I would think that it would simply have to stream large chunks of each drive in parallel, apply the parity coding and write out the stream of calculated blocks to the new disk.

Oh sure, it's a big linear streaming read/write - the issue is that if you're taking a 8, 12, 15, etc. hour window to rebuild, there's a fairly high chance of something else going wrong. Particularly because you have to worry that, eg, your flaky power source has been shortening drive lifetimes, that's why you saw the failure within the expected lifetime of the drive, and that's why the another one is going to die in…

Well, that sounds more like worrying about a single point of failure (e.g. that power source that can take out several drives at once) than what the redundancy is designed to protect against, i.e. uncorrelated failures of one of the drives in the array.

Of course having no redundancy at all during a rebuild that takes more than a negligible amount of time is still worrisome, but that's why we have multiple-redundancy raid schemes.

But you still have to untangle your concerns. A single point (or cause) of failure has to be handled differently than increasing probabilities of encountering independent UREs on separate drives during rebuild.

Re: HGST gets closer to shipping 10TB HDD

#13

Post by ghshephard » Fri, Mar 13, 2015, 2:36 AM UTC

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

This is probably why people like BackBlaze use RAIN (Redundant Array of Inexpensive Nodes) instead of RAID.

Also, their drive failure is 10 disks/day out of a population of 44,100 drives, about 0.02%.

So, between the lack of correlation of failure (their Nodes, or, "Pods" are in different racks), their ability to lose an entire POD regardless (they have 3 parity pods), and the relatively low disk failure rate - the large drives aren't a problem. A rebuild in 2-3 days is more than fine.

Re: HGST gets closer to shipping 10TB HDD

#14

Post by ghshephard » Fri, Mar 13, 2015, 2:37 AM UTC

moe wrote: post #8

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

Unless the root cause is something to do with the array - flakey controller, or power supply.

Re: HGST gets closer to shipping 10TB HDD

#15

Post by eck » Fri, Mar 13, 2015, 2:44 AM UTC

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

Well there's one issue that another whole drive will fail and you're screwed.

The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.

Re: HGST gets closer to shipping 10TB HDD

#16

Post by epistasis » Fri, Mar 13, 2015, 3:06 AM UTC

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to rebuild. (Though using erasure coding amplifies the amount of work to be done.)

Re: HGST gets closer to shipping 10TB HDD

#17

Post by ghshephard » Fri, Mar 13, 2015, 3:21 AM UTC

epistasis wrote: post #16

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to…

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes.

Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?

Re: HGST gets closer to shipping 10TB HDD

#18

Post by ghshephard » Fri, Mar 13, 2015, 3:23 AM UTC

eck wrote: post #15

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you're screwed. The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy dis…

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

Re: HGST gets closer to shipping 10TB HDD

#19

Post by chrisbolt » Fri, Mar 13, 2015, 3:27 AM UTC

ghshephard wrote: post #17

epistasis wrote: post #16

Earlier quoted context omitted.

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to…

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes. Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives…

> Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?

Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc).

http://ceph.com/docs/master/rados/operations/crush-map/#crus...

Re: HGST gets closer to shipping 10TB HDD

#20

Post by agumonkey » Fri, Mar 13, 2015, 3:35 AM UTC

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

I felt weird reading this. HDD are now for backup/capacity mostly, becoming CD-RW/Tapes of the day. Compared to recently shown Apple/Samsung SSDs going above 1GBps [1], feels even more out of place.

[1] http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-...