Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HGST gets closer to shipping 10TB HDD

zdnet.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: HGST gets closer to shipping 10TB HDD

#31

Post by gamegoblin »

ars wrote: post #2

Wow, hard drives are getting complicated! You have to write 256MB to change a single bit? And you have various zones, and you have to keep track of where data is written, because it's not written in order, and can be on any zone? You would need some sort of battery backed scratch/cache area to pull this off, so that you don't have to write very often, otherwise I can't imagine performance will be very good.

You only need to rewrite the whole zone if you want to modify something that's before the write pointer. That is, you can write data sequentially without penalty. It's modifying existing data that's the issue.

Re: HGST gets closer to shipping 10TB HDD

#32

Post by cdr »

agumonkey wrote: post #20
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

I felt weird reading this. HDD are now for backup/capacity mostly, becoming CD-RW/Tapes of the day. Compared to recently shown Apple/Samsung SSDs going above 1GBps [1], feels even more out of place. [1] http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-...

Apple must not be using the top-tier Samsung M.2 SSD; they can write at 1500MB/s+ and read at 2100MB/s. 1024MB/s is nothing. Now that SSDs are moving to using PCI-E lanes instead of SATA, we'll probably see quite a speed race. Really looking forward to the Intel 750, which will straight up be a PCI-E card for the desktop version.

http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-s...

Re: HGST gets closer to shipping 10TB HDD

#33

Post by ars »

pkaye wrote: post #24
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

I see a new feature for SATA called Rebuild Assist becoming available to help reduce the RAID recovery times.

http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC7...

The way it works is by realizing there are only a few bad sectors on the failed drive, so rebuild that area first, the used the "bad" drive to help rebuild the rest of the array at full speed.

Re: HGST gets closer to shipping 10TB HDD

#34

Post by jacquesm »

letstryagain wrote: post #28
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

We stopped using RAID5/6 several years ago because of rebuild time and associated performance drop. Now with these massive disks we don't even use RAID10 any more! Every disk is just paired with one mirror and that's it. (RAID1) The plan is to move to JBOD soon and just use software to store each file on 3 random disks across 3 separate servers.

That creates a substantial window during which you could lose the second drive.

Re: HGST gets closer to shipping 10TB HDD

#35

Post by cmurf »

walrus wrote: post #30
insaneirish wrote: post #27

Earlier quoted context omitted.

So, yea, Ceph: https://twitter.com/tbmcmullen/status/566856092185923584

Does this cause problems in practice? I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

Ceph isn't ready for production. GlusterFS is, and it also behaves more like a file system, while Ceph is more object storage. They serve different use cases but GlusterFS is more general purpose I think. No MDS required, scales fairly linearly. They're using it at Facebook with XFS and Btrfs underneath depending on workload. At home you can build little ARM gluster clusters, so it kinda even scales down! http://events.linuxfoundation.org/sites/events/files/slides/...

Re: HGST gets closer to shipping 10TB HDD

#36

Post by cmurf »

ghshephard wrote: post #18
eck wrote: post #15

Earlier quoted context omitted.

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you're screwed. The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy dis…

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.

Re: HGST gets closer to shipping 10TB HDD

#37

Post by nigwil_ »

cmurf wrote: post #35
walrus wrote: post #30

Earlier quoted context omitted.

Does this cause problems in practice? I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

Ceph isn't ready for production. GlusterFS is, and it also behaves more like a file system, while Ceph is more object storage. They serve different use cases but GlusterFS is more general purpose I think. No MDS required, scales fairly linearly. They're using it at Facebook with XFS and Btrfs underneath depending on workload. At home you can build little ARM gluster clusters, so it kinda even scales down! http://even…

"Ceph isn't ready for production." - this is not the case.

Ceph object-store known as RADOS is production ready and supported by Red Hat [1], in addition the block-device service known as RBD which is an access method to RADOS is also production ready and supported. RBD provides many SAN like features like snapshots, cloning, and soon mirroring. The CephFS (POSIX filesystem access method) is not yet a supported product but is used by several sites in production. Ceph also provides S3/Swift object-store gateway (radosgw) so existing applications can access the object store with a compatible API.

Ceph provides unified storage: providing object, block and filesystem access from a single cluster, so Ceph is the more general purpose technology. Ceph MDS provides metadata service for the CephFS POSIX-filesystem but is not needed if you're not using that feature. For development and testing Ceph will run on a single VM, or more realistically on 3 small VMs so the cluster characteristics can be explored.

[1] alternatively you can buy a production grade fully supported appliance product from Fujitsu based on Ceph known as ETERNUS CD10000. For most deployments Ceph runs well on commodity hardware and the Ceph community provides excellent support too.

Re: HGST gets closer to shipping 10TB HDD

#38

Post by morcheeba »

hyperion2010 wrote: post #7
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

Can't this problem be partially solved by having more read/write heads? I don't think it will fit in the same form factor, but if the limit is how fast you can spin the disk then having multiple independently controlled read/write heads per platter seems like one way out of them problem.

seagate dual head design: http://www.tomshardware.com/news/seagate-hdd-harddrive,8279....

Re: HGST gets closer to shipping 10TB HDD

#39

Post by DiabloD3 »

moe wrote: post #8
fiatmoney wrote: post #4

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing…

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

I have to say this is false. I work in that industry, and even when you have, say, a 4 drive RAID, and all 4 drives are the same model but intentionally different batches, you will still, more often than not, have 2 drive failures within 24 hours.

Now, what does this mean in, exactly? In that four drive scenario, RAID10 has a 1/3rd chance of failure (both failed drives being both the A or B parts of the inner RAID 0s is two out of six possible scenarios), RAID5 will fail no matter which two drives, RAID6 will survive no matter which two drives, ZFS's RAID-Z3 (three parity RAID5/6) will survive no matter which three drives, and RAID1 will survive no matter which three drives.

RAID10 is the most performant of the possible outcomes, and the usual makeup of a 4 drive array unless you absolutely need the storage, then its usually RAID5; unless you seriously do not care about sanity at all, then its RAID0 or just independent disks.

Now, my suggestion for 15 hour rebuild times? Whatever you do, have a hotspare so it can begin immediately rebuilding and do not use any RAID variant that can't handle more than 2 failures in 15 hours. This means no RAID10 or RAID5, only RAID6 or RAID-Z3.

Ceph mitigates this 15 hour time because it can rebuild a lost drive by just allocating new blocks on every other drive simultaneously and maxing out your network and/or storage IO (whichever is slowest) so the window of potential doom is much smaller (depending on Ceph cluster size, obviously, bigger the better in this case).

Re: HGST gets closer to shipping 10TB HDD

#40

Post by DiabloD3 »

craigyk wrote: post #23
ghshephard wrote: post #14

Earlier quoted context omitted.

Unless the root cause is something to do with the array - flakey controller, or power supply.

these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?

ZFS can mostly handle that. It will try to recover any data it can, and rebuild whatever it can. Due to how paranoid ZFS is designed, it will survive edge cases like this much better, which is also why I like deploying it here even at home.

And if it can't recover it? You can prove it by failed checksum check. This is why, in my opinion, ZFS is so much better than all other file systems: if it fails, you can prove it instead of just sorta vaguely questioning everything because you can't tell but you suspect it, and it slowly drives you insane.

Once you try ZFS, you never go back.

Post reply on HN