Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze hard drive reliability stats for Q3 2016

backblaze.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Backblaze hard drive reliability stats for Q3 2016

#71

Post by PuffinBlue »

atYevP wrote: post #69
PuffinBlue wrote: post #68

Earlier quoted context omitted.

Is that meant as a rude retort? Because if it is it from the team that currently can't keep a blog post online when you get a few thousand concurrent visitors, so you might keep yourself open to suggestions and perhaps undertake the BASIC best practices of keeping a Wordpress site up under load. If nothing else it shows a basic lack of planning for what you know to be a massively popular post, so turn a little of tha…

No, he was agreeing. We have a lot of projects on our map to shore up some of these types of issues, but our admins are in high demand, so some of the lower-priority tasks slip on occasion. Since we rarely have issues with the blog (today was an exception) it tends to be a "we know what we'd like to change, but we'll do it when we have time" type of silo on our website. *Edit -> to your above edit -> I think if you e…

Huh, it works! :-)

Re: Backblaze hard drive reliability stats for Q3 2016

#72

Post by drzaiusapelord »

pfarnsworth wrote: post #17

With hard drive size increasing so quickly but hard drive transfer speeds basically flat, I wonder if there are long-term implications for them with respect to recovery from backup and downtime. For example, if a whole rack goes down, and they are on 32TB drives in the future, for example, could it takes a week or more for their data to get online?

Heck, just a raid rebuild takes days nowadays if you dare use RAID5 still or, less risky, RAID6.

The smart move is to keep several smaller arrays instead of one big one. This lowers risk as well. I dont put in anything bigger than 7 disks into production. Past that I'm just asking for trouble. Its better to have 4 7 disk arrays than one 28 disk array. A drive fail means a quick rebuild and a restore is going to be 1/4 the time.

Re: Backblaze hard drive reliability stats for Q3 2016

#73

Post by kardos »

dom0 wrote: post #62
kardos wrote: post #57

Earlier quoted context omitted.

Is there a reason you couldn't use a second set of actuators?

There's kinda not enough space: https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Sa... You could probably do it if the platter diameter were reduced, which would then accordingly reduce capacity as well. At that point you could just as well use two drives and also get lower overall failure probability. Or you use SSD caches, or memory caches, ...

I was thinking of a Frankendrive with 5 actuators, which can do 5 parallel reads/writes, as long as they don't overlap. But the new dimensions would probably be the roadblock.

Re: Backblaze hard drive reliability stats for Q3 2016

#74

Post by distances »

atYevP wrote: post #14
liotier wrote: post #5

Earlier quoted context omitted.

> We likely won't have a Linux client for our backup service any time soon Ok. But why ? Technical obstacles such as having to deal with distribution diversity - or is it a way of market segmentation ?

It's a mixture of a couple things. One is that we tend to run pretty lean and our engineers are all booked up for the foreseeable future. Linux users are a passionate community, but we can't quite justify the development time for a market segment that is not very large. Additionally, because we run an unlimited model, a lot of people would immediately sign up and back up their Linux servers for $5/month and we'd sail…

It's worth considering though that the Linux compatibility is worth more than the actual market share. Our company (with roughly 90% Mac / 5% Linux / 5% Windows users) went with Crashplan to have a single backup solution for all of the employees.

Re: Backblaze hard drive reliability stats for Q3 2016

#75

Post by mahyarm »

dom0 wrote: post #40
pfarnsworth wrote: post #17

With hard drive size increasing so quickly but hard drive transfer speeds basically flat, I wonder if there are long-term implications for them with respect to recovery from backup and downtime. For example, if a whole rack goes down, and they are on 32TB drives in the future, for example, could it takes a week or more for their data to get online?

Well, uh, yes. No one really expects that rotational rust will get much faster, and in fact history shows that, compared to the increase in density, the increase in transfer rates are laughable at best. Between 1990 and today you are probably looking at a 20 000 times increase in density, yet transfer rates only increased around a factor of around 150-200. [In fact, from the early 1960 to today it's only a factor of…

How about the stupid idea of making a minimum block size be a multiple of the number of platters to write to. The block is divided evenly amongst all platters and are always at the same parallel track/sector. That way you can multiply the read/write speeds to be a function of the number of platters.

Instead of 50-100MB/s, you can get 4-8x the speed in large linear transfers, which helps get dead racks back up faster and would work quite well in backblaze's backup model.

Your block sizes will be huge, but I think in backblaze's case that doesn't really matter so much.

Re: Backblaze hard drive reliability stats for Q3 2016

#76

Post by Filligree »

toomuchtodo wrote: post #13
mixedCase wrote: post #11

Earlier quoted context omitted.

CrashPlan offers a Linux desktop client, and I believe they're a smaller player than Backblaze.

CrashPlan made a different decision. If I was to offer an unlimited backup service, I would offer it to novice users, not a small minority of power users and professionals who were going to break the business model with TBs of data. Let's be honest: when people see unlimited, most think "I don't have to worry about how much I'm storing" but a small group thinks "How can I take advantage of this?" Not supporting a Lin…

> Let's be honest: when people see unlimited, most think "I don't have to worry about how much I'm storing" but a small group thinks "How can I take advantage of this?"

I don't want to take advantage of it. I just happen to have 8TB of data to back up...

But backing up that much data over the internet isn't practical in any case.

Re: Backblaze hard drive reliability stats for Q3 2016

#77

Post by digler999 »

kardos wrote: post #73
dom0 wrote: post #62

Earlier quoted context omitted.

There's kinda not enough space: https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Sa... You could probably do it if the platter diameter were reduced, which would then accordingly reduce capacity as well. At that point you could just as well use two drives and also get lower overall failure probability. Or you use SSD caches, or memory caches, ...

I was thinking of a Frankendrive with 5 actuators, which can do 5 parallel reads/writes, as long as they don't overlap. But the new dimensions would probably be the roadblock.

I was going to ask this too, but start with a simple 2nd voice coil on the opposite side of the spindle, keep the 3.5" form factor and just make it longer. My guess is the motion of one actuator would disrupt the airflow of the other, since we're talking about micrometer-ish (?) mechanical tolerances. This might be able to be mitigated by interleaving which side each arm wrote to. So the left arm's heads address the tops of the platters, the right side addresses the bottoms.

Perhaps another option is to go back to the 1980's hdd designs where the arm moves straight down the radius of the platter. This design might permit multiple heads on the same arm. I'm sure all this stuff has been researched thoroughly.

Either way, this doubles/triples the probability of mechanical failure

Re: Backblaze hard drive reliability stats for Q3 2016

#78

Post by cm2187 »

drzaiusapelord wrote: post #72
pfarnsworth wrote: post #17

With hard drive size increasing so quickly but hard drive transfer speeds basically flat, I wonder if there are long-term implications for them with respect to recovery from backup and downtime. For example, if a whole rack goes down, and they are on 32TB drives in the future, for example, could it takes a week or more for their data to get online?

Heck, just a raid rebuild takes days nowadays if you dare use RAID5 still or, less risky, RAID6. The smart move is to keep several smaller arrays instead of one big one. This lowers risk as well. I dont put in anything bigger than 7 disks into production. Past that I'm just asking for trouble. Its better to have 4 7 disk arrays than one 28 disk array. A drive fail means a quick rebuild and a restore is going to be 1/…

Stupid question but why does it take more time to rebuild a 10 disks array than a 6 disks array. I mean modern PCIe can go several GB/s and the disks are in parallel, so it should take the same time to rebuild irrespective of how many disks.

Re: Backblaze hard drive reliability stats for Q3 2016

#79

Post by SteveNuts »

brianwski wrote: post #54
fpp wrote: post #43

Earlier quoted context omitted.

I'll see from your reports that you're migrating 1000s of 2TB drives to 8TB drives - what is actually happening to your old 2TB drives? Guess you are not throwing those away so what are you doing with them.

Disclaimer: I work at Backblaze. We securely wipe the drives, then we sell them to a "used hard drive reseller".

I know it's a lot easier to wholesale resell them, but it'd be so awesome if you could dedicate a portion to be sold to the homelabbers among us.

Re: Backblaze hard drive reliability stats for Q3 2016

#80

Post by atYevP »

PuffinBlue wrote: post #71
atYevP wrote: post #69

Earlier quoted context omitted.

No, he was agreeing. We have a lot of projects on our map to shore up some of these types of issues, but our admins are in high demand, so some of the lower-priority tasks slip on occasion. Since we rarely have issues with the blog (today was an exception) it tends to be a "we know what we'd like to change, but we'll do it when we have time" type of silo on our website. *Edit -> to your above edit -> I think if you e…

Huh, it works! :-)

#SubvertingTheInternetsSince1998 :D
Post reply on HN