The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story...
I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB:
pool: tank
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub repaired 4K in 21h05m with 0 errors on Mon Aug 29 20:52:45 2016
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ada3 ONLINE 0 0 2
I monitored the situation. But every week, subsequent scrubs would continue to find errors on ada3, and on more data (100-5000kB):
2016-09-05: 1.7MB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-09-12: 5.2MB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-09-19: 300kB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-09-26: 1.8MB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-10-03: 3.1MB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-10-10: 84kB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-10-17: 204kB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-10-24: 388kB silently corrupted on ada3 (ST5000DM000-1FK178)
2016-11-07: 3.9MB silently corrupted on ada3 (ST5000DM000-1FK178)
The next week. The server became unreachable during a scrub. I attempted to access the console over IPMI but it just showed a blank screen and was unresponsive. I rebooted it.
The next week the server again became unreachable during a scrub. I could access the console over IPMI but the network seemed non-working even though the link was up. I checked the IPMI event logs and saw multiple correctable memory ECC errors:
Correctable Memory ECC @ DIMM1A(CPU1) - Asserted
The kernel logs reported muliple Machine Check Architecture errors:
MCA: Bank 4, Status 0xdc00400080080813
MCA: Global Cap 0x0000000000000106, Status 0x0000000000000000
MCA: Vendor "AuthenticAMD", ID 0x100f80, APIC ID 0
MCA: CPU 0 COR OVER BUSLG Source RD Memory
MCA: Address 0x5462930
MCA: Misc 0xe00c0f2b01000000
At this point I could not even reboot remotely the server via IPMI. Also, I theorized that in addition to
correctable memory ECC errors, maybe the DIMM experienced
uncorrectable/undetected ones that were really messing up the OS but also IPMI. So I physically removed the module in "DIMM1A", and the server has been working perfectly well since then.
The reason these memory errors always happened on ada3 is not because of a bad drive or bad cables, but likely due to the way FreeBSD allocates buffer memory to cache drive data: the data for ada3 was probably located right on defective physical memory page(s), and the kernel never moves that data around. So it's always ada3 data that seems corrupted.
PS: the really nice combinatorial property of raidz2 with 6 drives is that when silent corruption occurs, the kernel has 15 different ways to attempt to rebuild the data ("6 choose 4 = 15").