I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…
Figure this is as good of a time as any to ask this: There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?
Bit flips (for all reasons) occur in buses, registers, caches, etc. Anything that has state can have state changed incorrectly.
This is why filesystems like ZFS exist and storage formats have pervasive checksums.