Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What SMART Stats Tell Us About Hard Drives

backblaze.com

Post reply on HN
Print view View original

31–40 of 71 posts

Re: What SMART Stats Tell Us About Hard Drives

#31

Post by Retric »

sn wrote: post #26
Retric wrote: post #8

Earlier quoted context omitted.

Thresholds are often useful with these kinds of stats. Aka a drive moving one sector might mean nothing, but moving 30 in a week could be great predictor. Further they only have 70k drives across a range of product lines so what predicts drive X failing very well might say little about drive Y. PS: Rememebr all RAM gets bit flit errors over time. Which is one of the reasons rebooting is often so useful, but also mean…

ECC ram will either correct the error or can raise a non maskable interrupt if it can't be corrected.

ECC vastly lowers, but does not remove this problem. You can for example get bit flip errors while doing the checksum.

As a personal user it's a non issue but scale things to ~70k devices and you get ~1.7 million device hours per day.

Re: What SMART Stats Tell Us About Hard Drives

#32

Post by ekianjo »

esaym wrote: post #14

Ten years ago when I was I trying to learn how to "program", I wrote this bash script (to be added into /etc/cron.daily) that dumps a few smart stats that are normally 0 or slow changing, diffs it with the copy from the previous run, and if anything is different (and cron in configured right) it will email you the diff. Every linux machine I touch gets this file dropped onto it. I've replaced many hard drives because…

Did you keep any data on how often you got to replace your drives because of such signs ?

Re: What SMART Stats Tell Us About Hard Drives

#33

Post by koolba »

esaym wrote: post #14

Ten years ago when I was I trying to learn how to "program", I wrote this bash script (to be added into /etc/cron.daily) that dumps a few smart stats that are normally 0 or slow changing, diffs it with the copy from the previous run, and if anything is different (and cron in configured right) it will email you the diff. Every linux machine I touch gets this file dropped onto it. I've replaced many hard drives because…

(Warning pedantic script review)

Add a "set -e" to catch errors. Say if the disk can't be read or file can't be written.

Why reuse the same temp file? Make a new one with mktemp and auto clean it via an exit trap. As it's written this isn't concurrently safe.

Exiting -1 on error? Don't use negatives.

Wrap it all in a main() function and use locals instead of global vars.

Re: What SMART Stats Tell Us About Hard Drives

#34

Post by arielhn »

teh_klev wrote: post #30
arielhn wrote: post #17

Small nitpicking here, but the moment it popup a modal dialog asking me to enter my email for some kind of subscription I simply close the tab. I do this since three months ago for any unknown website I visit. Such nuisance for what might be a good read.

Shame...you could've just dismissed the popup and not missed out on an interesting article, it's same energy expended but with a nett gain instead of your loss. A small price to pay for BackBlaze willing to share interesting stuff like this and hardly the most egregious examples of this type of thing. Also these types of complaints have been done to death here over the years and are really, really tedious. Please com…

First of all, I was on a public transport when I click on that link, my 'consuming' experience already not optimal from the point of view of readership. Many technical people, like I do, are busy people with short tolerance on things that detract from what I'm supposed to read or comprehend. Unless I can just read right there right away, I'm just going to skip to the next tab.

Secondly, I noticed that this is a trend right now; where you get to a page and after a few seconds a dialog just thrown into your face with little disregard to you (the reader) is trying to concentrate to read the content. To me that is rude, you don't go to a bookstore while reading the table of content a salesman grab that book from you and tell you "would you like me to take your email address so that we can notify you when we have new books available?" without wondering what kind of establishment that allow this kind of behavior.

Third, I got the link from HN it was easier for me to go back to this tab, login, hit reply than registering a disqus account and then enter a comment there.

With that said, I dont want to blog about this on Medium or whatever, I dont need clicks by moaning about every little things, this is my way of protesting on what I perceive is happening right now and that's why I start with "small nitpicking".

Re: What SMART Stats Tell Us About Hard Drives

#35

Post by sn »

Retric wrote: post #31
sn wrote: post #26

Earlier quoted context omitted.

ECC ram will either correct the error or can raise a non maskable interrupt if it can't be corrected.

ECC vastly lowers, but does not remove this problem. You can for example get bit flip errors while doing the checksum. As a personal user it's a non issue but scale things to ~70k devices and you get ~1.7 million device hours per day.

Fair enough. Your original comment did not make this distinction.

I dug and found https://www.fiala.me/pubs/papers/sc12-redmpi.pdf the title of which is "Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing."

They state that for a cray there was a double bit flip about 1x/day for 75k modules. To (probably incorrectly) extrapolate, if you have a server with 16 modules that would be equivalent to a single double failure about once every 13 years.

Re: What SMART Stats Tell Us About Hard Drives

#37

Post by tedunangst »

Isn't the reverse stat more interesting? What percentage of drives reporting an error fail within X weeks? I don't want to know how many failed drives had errors, I want to know how many errored drives fail. (A more accurate title might be "What failed drives tell us about SMART stats".)

Re: What SMART Stats Tell Us About Hard Drives

#38

Post by jlarocco »

teh_klev wrote: post #30
arielhn wrote: post #17

Small nitpicking here, but the moment it popup a modal dialog asking me to enter my email for some kind of subscription I simply close the tab. I do this since three months ago for any unknown website I visit. Such nuisance for what might be a good read.

Shame...you could've just dismissed the popup and not missed out on an interesting article, it's same energy expended but with a nett gain instead of your loss. A small price to pay for BackBlaze willing to share interesting stuff like this and hardly the most egregious examples of this type of thing. Also these types of complaints have been done to death here over the years and are really, really tedious. Please com…

On the contrary, those popups are a really stupid and annoying trend, and I think it's good people voice their dissatisfaction with them.

Without people leaving the sites and complaining on places like HN, web designers will have no feedback that it's such a stupid idea.

Re: What SMART Stats Tell Us About Hard Drives

#39

Post by caf »

koolba wrote: post #33
esaym wrote: post #14

Ten years ago when I was I trying to learn how to "program", I wrote this bash script (to be added into /etc/cron.daily) that dumps a few smart stats that are normally 0 or slow changing, diffs it with the copy from the previous run, and if anything is different (and cron in configured right) it will email you the diff. Every linux machine I touch gets this file dropped onto it. I've replaced many hard drives because…

(Warning pedantic script review) Add a "set -e" to catch errors. Say if the disk can't be read or file can't be written. Why reuse the same temp file? Make a new one with mktemp and auto clean it via an exit trap. As it's written this isn't concurrently safe. Exiting -1 on error? Don't use negatives. Wrap it all in a main() function and use locals instead of global vars.

You don't even need the 'smartStats' temporary file at all, since you can do that with just one grep:

  smartctl -a /dev/sda |
    grep '\(Reallocated_Sector_Ct\|Current_Pending_Sector\|Offline_Uncorrectable\|UDMA_CRC_Error_Count\)' > /root/stats

Re: What SMART Stats Tell Us About Hard Drives

#40

Post by jimmaswell »

koolba wrote: post #33
esaym wrote: post #14

Ten years ago when I was I trying to learn how to "program", I wrote this bash script (to be added into /etc/cron.daily) that dumps a few smart stats that are normally 0 or slow changing, diffs it with the copy from the previous run, and if anything is different (and cron in configured right) it will email you the diff. Every linux machine I touch gets this file dropped onto it. I've replaced many hard drives because…

(Warning pedantic script review) Add a "set -e" to catch errors. Say if the disk can't be read or file can't be written. Why reuse the same temp file? Make a new one with mktemp and auto clean it via an exit trap. As it's written this isn't concurrently safe. Exiting -1 on error? Don't use negatives. Wrap it all in a main() function and use locals instead of global vars.

It's just a small bash script, and one that's apparently worked well for 10 years. Rewriting it to J2EE standards would just be a waste of time; the best outcome is that it still works the same, and the other outcome is that you introduced a new bug refactoring it.
Post reply on HN