Hard Drive Reliability Update – Sep 2014 • Hacker News Board

Re: Hard Drive Reliability Update – Sep 2014

#161

Post by disordr » Wed, Sep 24, 2014, 11:03 PM UTC

gioele wrote: post #144

disordr wrote: post #77

I really want to applaud backblaze for publishing these reports and stats. Too many companies closely guard this information that really helps the larger community. Based on the previous blogs from backblaze, when I built out our new hadoop cluster, I purchased 1450 Hitachi drives. I plan to gather our failure rates and publish them as backblaze does. Thanks for blazing the path!

> I purchased 1450 Hitachi drives. Isn't using very similar drives a problem because their failure rates are not statistically independent, so there is an high probability that they will all fail at the same time?

Yes, there is a concern in getting a bad batch, or having several drives fail all at once. Interestingly enough, while opening those 1450 anti-static cases and seeing the manufacture date of them, they were in separate batches (some with more dust on the cases than others), so we have some heterogeneous-ness on them. HDFS with its various levels of replication factors, and our backups/DR for high-value data takes care of the rest.

Re: Hard Drive Reliability Update – Sep 2014

#162

Post by hga » Thu, Sep 25, 2014, 12:56 PM UTC

ShinyObject wrote: post #160

ars wrote: post #114

No Toshiba hard disks apparently. HGST and Wester Digital are the same company, but it seems they have separate product lines? It's confusing.

The Hitachi drives are Toshiba drives. The 3.5" desktop and server Hitachi drives are all manufactured by Toshiba. WD wanted to buy Hitachi's drive division so there would be a WD/Seagate drive duopoly, European regulators said they had to sell Hitachi's 3.5" to someone else before they would approve the merger. Toshiba stepped up and got it. http://www.anandtech.com/show/5635/western-digital-to-sell-h... So yes it i…

There's a much less sinister reason for WD buying HGST: SCSI/SAS support. Way back when in the '90s WD tried to enter that market, and a company I was at even got some of those drives, but from what I heard while their read rate was OK their write rates weren't competitive with Seagate and IBM (which sold their disk drive business to Hitachi after the Deathstar screwup). And they exited that segment of the business, and hadn't, last time I checked, returned to it.

Re: Hard Drive Reliability Update – Sep 2014

#163

Post by GravityWell » Thu, Sep 25, 2014, 1:04 PM UTC

This is extraordinarily useful and unique. My compliments to Backblaze for making this available. This is the type of empirical data I would love to have for as many things as possible: SSDs, monitors, TVs, kitchen appliances, tablets, cars, etc.

Re: Hard Drive Reliability Update – Sep 2014

#164

Post by blahblah234 » Thu, Sep 25, 2014, 4:27 PM UTC

ShinyObject wrote: post #160

ars wrote: post #114

No Toshiba hard disks apparently. HGST and Wester Digital are the same company, but it seems they have separate product lines? It's confusing.

The Hitachi drives are Toshiba drives. The 3.5" desktop and server Hitachi drives are all manufactured by Toshiba. WD wanted to buy Hitachi's drive division so there would be a WD/Seagate drive duopoly, European regulators said they had to sell Hitachi's 3.5" to someone else before they would approve the merger. Toshiba stepped up and got it. http://www.anandtech.com/show/5635/western-digital-to-sell-h... So yes it i…

Toshiba drives are not exactly Hitachi drives.

First, Toshiba had their own 3.5" drives long before the merger with Hitachi. For example, Toshiba MK2002TSKB 2TB 3.5" hard drive was on sale since 2011.

Toshiba's own design and factories wouldn't magically disappear after acquiring Hitachi's assets. So, after the merger, Toshiba sells some ex-Hitachi drives, for example toshiba DT01ACA300 3TB is obviously a relabeled Hitachi drive. But I believe that Toshiba MD03ACA300 3TB and Toshiba MD04ACA300 3TB drives are based on Toshiba's own design because they don't look like Hitachi or DT01ACA300. I suppose reliability and performance will be different between these three Toshiba 3TB drives. And it would be interesting to get some info on this.

Also, HGST is wholly owned by Western Digital. Despite regulators requirement, they didn't sell all 3.5" assets to Toshiba, only some of them. Most of the good stuff (that is, everything currently sold under HGST brand) still went to WD. So, even ex-Hitachi drives sold under Toshiba brand (by Toshiba) and under HGST brand (by WD) might have different reliability.

Re: Hard Drive Reliability Update – Sep 2014

#165

Post by brianwski » Fri, Sep 26, 2014, 3:07 AM UTC

jamroom wrote: post #98

sauere wrote: post #94

Hard drive age a bad parameter to use. It should be the hours the drive was actually powered on.

I have a feeling that with Backblaze that ALL of their drives are mostly powered on continuously until they fail (i.e. they might sit for a bit at their data center waiting to go into one of their pods, but after that they pretty much run until they fail).

Backblaze Employee here - this is correct. Pods get powered up, then pummeled with customer data until full. Then they stay spinning forever, but the rate of writes drops to "churn" rates for the rest of the drive's life. This use pattern is all we have experience with, so if your application is different (maybe a database that gets heavy load forever) then your outcomes may vary.

Re: Hard Drive Reliability Update – Sep 2014

#166

Post by andrewaylett » Sun, Sep 28, 2014, 1:00 PM UTC

cjensen wrote: post #106

andrewaylett wrote: post #92

Earlier quoted context omitted.

It sounds like you might not have TRIM enabled? Or am I misunderstanding what you mean by 'usage'?

I'm talking about the "endurance" spec. Because Flash Memory can only be erased a finite number of times[1], SSD manufacturers specify the drive's endurance -- defined as the number of bytes which can be written to the SSD over its lifetime. The SSDs helpfully keep track of how many bytes you've written and report that in the SMART info. For example, on my Windows dev system, HDD Guardian reports that I've used up 42…

That makes sense, thank you for explaining it to me.

Re: Hard Drive Reliability Update – Sep 2014

#167

Post by linker3000 » Thu, Oct 02, 2014, 10:14 PM UTC

Someone1234 wrote: post #155

linker3000 wrote: post #120

Earlier quoted context omitted.

It's now called "HGST (a Western Digital company)" and the two product lines (WD and HGST HDDs) are totally separate in both parts and manufacturing. /HGST employee

You don't work in the HDD division, right? Someone seriously needs to look into ADM (advanced power management) on your 2.5" drives. It is currently set too low and the drives power down instantly after each instruction, resulting in simply astonishingly terrible IO (and seriously lag spikes). It is so bad even running HD video off of some 2014 HGST laptop hard drives causes distortion. There's also no way to increas…

Nope, I do other stuff with Flash Platforms. Go through the support link on the web site and send in your comments.

Re: Hard Drive Reliability Update – Sep 2014

#168

Post by linker3000 » Thu, Oct 02, 2014, 10:17 PM UTC

Jetlag wrote: post #151

linker3000 wrote: post #120

Earlier quoted context omitted.

It's now called "HGST (a Western Digital company)" and the two product lines (WD and HGST HDDs) are totally separate in both parts and manufacturing. /HGST employee

Can you explain what "assets" Toshiba got to start their 3.5" drive business? Are HGST and Toshiba making very similar 3.5" drives?

Can't help with that one at all.