Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HGST gets closer to shipping 10TB HDD

zdnet.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: HGST gets closer to shipping 10TB HDD

#21

Post by epistasis »

ghshephard wrote: post #17
epistasis wrote: post #16

Earlier quoted context omitted.

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to…

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes. Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives…

Yes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week.

And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.

Re: HGST gets closer to shipping 10TB HDD

#22

Post by vidarh »

moe wrote: post #8
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller.

(We didn't lose any data - everything is replicated elsewhere in addition to the raid)

Re: HGST gets closer to shipping 10TB HDD

#23

Post by craigyk »

ghshephard wrote: post #14
moe wrote: post #8

Earlier quoted context omitted.

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

Unless the root cause is something to do with the array - flakey controller, or power supply.

these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?

Re: HGST gets closer to shipping 10TB HDD

#24

Post by pkaye »

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

I see a new feature for SATA called Rebuild Assist becoming available to help reduce the RAID recovery times.

Re: HGST gets closer to shipping 10TB HDD

#25

Post by ryan-c »

ghshephard wrote: post #18
eck wrote: post #15

Earlier quoted context omitted.

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you're screwed. The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy dis…

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively.

http://www.enterprisestorageforum.com/storage-hardware/selec...

Re: HGST gets closer to shipping 10TB HDD

#26

Post by ryan-c »

ars wrote: post #2

Wow, hard drives are getting complicated! You have to write 256MB to change a single bit? And you have various zones, and you have to keep track of where data is written, because it's not written in order, and can be on any zone? You would need some sort of battery backed scratch/cache area to pull this off, so that you don't have to write very often, otherwise I can't imagine performance will be very good.

Having maybe 16 to 64 GB of flash storage (for writeback cache) would probably make a significant difference here. I think over the next five to ten years tiered storage will probably go mainstream.

Re: HGST gets closer to shipping 10TB HDD

#27

Post by insaneirish »

epistasis wrote: post #16
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to…

So, yea, Ceph: https://twitter.com/tbmcmullen/status/566856092185923584

Re: HGST gets closer to shipping 10TB HDD

#28

Post by letstryagain »

fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

We stopped using RAID5/6 several years ago because of rebuild time and associated performance drop. Now with these massive disks we don't even use RAID10 any more! Every disk is just paired with one mirror and that's it. (RAID1)

The plan is to move to JBOD soon and just use software to store each file on 3 random disks across 3 separate servers.

Re: HGST gets closer to shipping 10TB HDD

#29

Post by letstryagain »

moe wrote: post #8
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

At scale everything happens, even very rare events.

Re: HGST gets closer to shipping 10TB HDD

#30

Post by walrus »

insaneirish wrote: post #27
epistasis wrote: post #16

Earlier quoted context omitted.

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to…

So, yea, Ceph: https://twitter.com/tbmcmullen/status/566856092185923584

Does this cause problems in practice?

I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

Post reply on HN