Ten years ago when I was I trying to learn how to "program", I wrote this bash script (to be added into /etc/cron.daily) that dumps a few smart stats that are normally 0 or slow changing, diffs it with the copy from the previous run, and if anything is different (and cron in configured right) it will email you the diff. Every linux machine I touch gets this file dropped onto it. I've replaced many hard drives because…
(Warning pedantic script review) Add a "set -e" to catch errors. Say if the disk can't be read or file can't be written. Why reuse the same temp file? Make a new one with mktemp and auto clean it via an exit trap. As it's written this isn't concurrently safe. Exiting -1 on error? Don't use negatives. Wrap it all in a main() function and use locals instead of global vars.
What SMART Stats Tell Us About Hard Drives
61–70 of 71 posts
Re: What SMART Stats Tell Us About Hard Drives
#62Might be a good time to plug my little baby - https://diskovery.io If you want to have a quick, but in-depth look at your drives, it'll give you lots of data, including the SMART table interpreted in a vendor-specific way. It also understands some RAID setups, and more support for this is upcoming. Windows only, at the moment. To explain a bit of a context - SMART data comprises a set of attributes and each attribute…
Has anyone ported your work to Linux or MacOS? I guess not since iIt looks like it would be very OS specific. It looks like an incredible tool.
It is indeed pretty OS specific.
Not the SMART part, but how you talk to the drives and controllers and how storage is generally sliced into partitions, volumes, etc. Windows has a fairly comprehensive version of Software RAID, but in true Microsoft fashion they do things ass-backwards in more than one place. For example, striped volumes (RAID 0) will use only a part of a partition for each stripe, but to learn that you'd have to talk to Virtual Disk Service rather than regular Disk/Volume management API. This is, basically, as unportable as it gets.
Re: What SMART Stats Tell Us About Hard Drives
#63Earlier quoted context omitted.
If the presence of smart counters indicates imminent failure, their absence by definition indicates health. Not perfectly, obviously, but this is about probabilities. Say you want some level of data security (i. e. 99.9% over one year). The formula for the risk of data loss is (r^n) where r is the failure rate of drives and n is the number of drives (assuming independence and using just mirroring). But – if you have…
This is a discrete problem. You cannot have 2.9 replicas instead of 3. The fact that 24% of these drives failed without smart indicators means no test exists which can lower your replication from 3 to 2.
You can reduce your risk by requiring that no more than one of the three replicas is on an iffy drive, creating new replicas on healthy drives if that's not true.
Re: What SMART Stats Tell Us About Hard Drives
#64SMART is a fantastic exercise in sensitivity and specificity. As backblaze is showing with this data, SMART stats have poor sensitivity, but what's much worse for those who run big fleets of drives is their poor specificity. Lots of healthy drives are reported unhealthy by SMART. If I'm running a gold-plated database server, that doesn't matter. A couple of extra planned drive replacements is a small price to pay for…
It's unfair to blame SMART stats for this. On their own, SMART does not declare a drive 'healthy' or 'unhealthy', it's up to the user to decide. Each stat on its own seems very specific, e.g. Number of reallocated sectors. I'm not sure how you think this is 'poor specificity', the stats are measuring one thing exactly, you can't get more specific than that!
Re: What SMART Stats Tell Us About Hard Drives
#65SMART is a fantastic exercise in sensitivity and specificity. As backblaze is showing with this data, SMART stats have poor sensitivity, but what's much worse for those who run big fleets of drives is their poor specificity. Lots of healthy drives are reported unhealthy by SMART. If I'm running a gold-plated database server, that doesn't matter. A couple of extra planned drive replacements is a small price to pay for…
It's unfair to blame SMART stats for this. On their own, SMART does not declare a drive 'healthy' or 'unhealthy', it's up to the user to decide. Each stat on its own seems very specific, e.g. Number of reallocated sectors. I'm not sure how you think this is 'poor specificity', the stats are measuring one thing exactly, you can't get more specific than that!
Actually, it does declare. Direct quote from smartctl output: "SMART overall-health self-assessment test result: PASSED"
Re: What SMART Stats Tell Us About Hard Drives
#66Earlier quoted context omitted.
(Warning pedantic script review) Add a "set -e" to catch errors. Say if the disk can't be read or file can't be written. Why reuse the same temp file? Make a new one with mktemp and auto clean it via an exit trap. As it's written this isn't concurrently safe. Exiting -1 on error? Don't use negatives. Wrap it all in a main() function and use locals instead of global vars.
It's just a small bash script, and one that's apparently worked well for 10 years. Rewriting it to J2EE standards would just be a waste of time; the best outcome is that it still works the same, and the other outcome is that you introduced a new bug refactoring it.
> It's just a small bash script, and one that's apparently worked well for 10 years. Rewriting it to J2EE standards would just be a waste of time; the best outcome is that it still works the same, and the other outcome is that you introduced a new bug refactoring it.
I don't see how following best practices for bash scripting (or really shell scripting in general) can be compared to J2EE standards.
Seek quality in all your scripting so it becomes the norm. Otherwise you'll end up having crap like that in something mission critical.
Re: What SMART Stats Tell Us About Hard Drives
#67I get a tingling that Bayes theorem applied to these probabilities could lead to some insights. Might do this later if I find the ambition.
Re: What SMART Stats Tell Us About Hard Drives
#68Earlier quoted context omitted.
There aren't too many vendors for spinning disks, and if you have a lot of disks it doesn't take too long to see that the sector count metrics correspond to sectors. In my experience, bad sector count is a good predictor of future trouble, and running disks until they threw read errors (before we were running smart monitoring), they all had lots of bad sectors. That said, there's a threshold, getting to 100 slowly is…
"SSDs though, they just disappear from the bus when they fail" This isn't always true, and actually shouldn't ever be true - it's a particular failure mode you're seeing, and while it appears to be one common across a number of SSD controllers, it's still a pretty sorry fact that it happens. All SSDs (at least all not-complete-rubbish ones) report some kind of flash/media wearout indicator via SMART, which isn't nece…
With thousands of sata SSDs, I've seen one fail in a traditional fashion (some sectors weren't readable, otherwise mostly fine) and the rest of the maybe hundred that failed would just disappear from the bus. I don't monitor the wear out indicators, but from occasional looking, we're never near a significant fraction of the wear capacity. I'm very happy not to have anymore spinning disks in production, because the ssds fail less often, it's just the failures are more annoying, because it's hard to have an orderly shutdown when disks disappear.
Re: What SMART Stats Tell Us About Hard Drives
#69First these counters vary in meaning and support by vendor/model, and it would be nice if someone were to come along and mandate further standardized ones. Instead you have to tune everything for each drive model. In this regard SCSI is a little better (more on that later).
Second, timeouts and uncorrectable errors are generally being reported to the controller as part of normal operation. So having SMART tracking them is just a bonus. Either of those two conditions is usually sufficient to kick a drive out of a functional RAID array because those are data loss events. Most drives have layers and layers of ECC, so in order to get an uncorrectable error a lot of bits need to be flipped in the target sector. For that to happen it likely indicates there is something mechanical going on which is likely to affect adjacent tracks/sectors. Of course if you never scrub your drives its possible bitrot accumulates on a perfectly functional device until sectors aren't recoverable.
In my previous life I found it much more interesting to track the rate of soft error counts during scrub operations. Particularly, in larger arrays because sometimes a drive would start getting slower (which is frequently caused by read retries in the drive itself or problems tracking the embedded servo/etc) and the correctable error counts would start to steadily rise followed by actual timeouts/uncorrectable errors. Of course these days, it seems most drives won't show the correctable error counts because it would freak people out. Instead you have to infer it from seek errors and relocated sector counts. Although, it might now be considered a SAS/SATA differentiator. SCSI has standardized log pages with more detailed information. (random google hit http://www.seagate.com/staticfiles/support/disc/manuals/scsi... page 238) Note the errors are categorized as corrected without delay, with substantial delay, and corrected on a retry. By comparison the SMART data isn't particularly "smart".
Re: What SMART Stats Tell Us About Hard Drives
#70Earlier quoted context omitted.
This is basically what smartmontools does: /var/log/syslog:Oct 6 08:14:10 hostname smartd[573]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 62 /var/log/syslog:Oct 6 09:44:10 hostname smartd[573]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 62 to 61 /var/log/syslog:Oct 6 10:14:10 hostname smartd[573]: Device: /dev/sda [SAT],…
I respectfully disagree. Judging from the output you posted, it just spams logfiles with irrelevant info, training the user to ignore it, so they will never notice when important values change in an interesting way.
Really?
My hard disks are fine, so right now the only stat changing is that, so that's what `rgrep smartd /var/log` turned up, so that's what I posted.
Did you consider looking at `man smartd.conf`? Did you consider that reporting and logging the temperature is entirely optional? Did you consider that you can configure it to log and report and email warnings for exactly the stats that you want? Did you consider that, for example, the stock Debian/Ubuntu package will pop up a window in your X session when stats which actually indicate potential failure change, like sector reallocations, read errors, etc?
Nah, just jump to the conclusion that all it does is spam logs with temperature readings, and downvote away!