I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…
The last time I did research into this, this was the most concrete real world example I could find, which works out to ~41 bitflips/GiB-month: > Jaguar had 360 terabytes of main memory [and] was logging ECC errors at a rate of 350 per minute Source: https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-...
What Flips Your Bit: Cosmic Ray Errors at Mozilla
81–90 of 96 posts
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#82We see a correlation between (major) solar activity and hash/signature verification failures from clients -- on the order of millions of verifications per day, only 30k failures per day, max.
I just finished looking into it in our reporting and was pretty impressed to see spikes lineup with dates here: https://www.spaceweatherlive.com/en/solar-activity/top-50-so...
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#83Earlier quoted context omitted.
This is tangential, but "Cosmic rays do not cause bit flips at a significant rate" is the default position and does not need supporting evidence. It's like if I claimed that planes occasionally crash due to a local density of dark matter pulling them down. Nobody would need to provide evidence against my theory, it's outrageous. I'm the one who has to provide the evidence.
> This is tangential, but "Cosmic rays do not cause bit flips at a significant rate" is the default position No it's not. It's very well accepted in the silicon industry that cosmic ray events (the secondary particles) are responsible for enough bit flips that they have to be concerned with designing robust circuits and logic to achieve desired failure rates. https://www.microsemi.com/document-portal/doc_view/130760-…
Here’s contradicting evidence to your position:
https://static.googleusercontent.com/media/research.google.c...
The point op makes is that the more complicate a claim is made, the more evidence is required. More common sources of errors would seem to be more likely and thus more common causes of bit flips.
Thus more evidence is required for the cosmic ray hypothesis being a dominant reason than anything else. We know that empirically there’s ~1 bug in every 1k lines of code. 1 in 10k if you have very good tests. But flip type errors are probably less common so let’s guess and say 1 in 10 million. There’s about ~30 million lines of code in the Linux kernel. There’s probably a similar amount of userspace code (eg Firefox is also around 20 million lines). Then think about the Verilog that backs HW designs. I don’t know the size of those codebases to have estimates but it feels like bit flip bugs are possible there. Then you’ve got to actually synthesize that digital logic and implement it in analog space. Components could easily be driven out of spec electrically (whether by accident, manufacturing defect, or swapping in lower cost components) and bit flips would be comparatively a common type of error when shuttling them around, especially sensitive across high bandwidth links that aren’t error-checked.
The point is, the combined probability of all these sources of errors seems higher probability than true cosmic rays being behind bit flips. The Google paper is just more evidence of this. I’m sure measuring just for cosmic rays you’ll be able to see their impact. In a running production at scale on variable quality hardware running on arbitrary software versions, all other sources of errors would seem like more likely first order effects that would swamp any ability to detect cosmic rays. Not to say that Mozilla hasn’t accounted for it. Just that OP’s position is the default sensible position to start from (ie Occam’s razor).
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#84Earlier quoted context omitted.
> This is tangential, but "Cosmic rays do not cause bit flips at a significant rate" is the default position No it's not. It's very well accepted in the silicon industry that cosmic ray events (the secondary particles) are responsible for enough bit flips that they have to be concerned with designing robust circuits and logic to achieve desired failure rates. https://www.microsemi.com/document-portal/doc_view/130760-…
Here’s contradicting evidence to your position: https://static.googleusercontent.com/media/research.google.c... The point op makes is that the more complicate a claim is made, the more evidence is required. More common sources of errors would seem to be more likely and thus more common causes of bit flips. Thus more evidence is required for the cosmic ray hypothesis being a dominant reason than anything else. We know…
That's not contrasting evidence. Defects are certainly common sources of error, particularly with cheap commodity components like those used in google's fleet. That does not prove cosmic rays aren't a significant source of SEUs [in any computing device].
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#85Earlier quoted context omitted.
Here’s contradicting evidence to your position: https://static.googleusercontent.com/media/research.google.c... The point op makes is that the more complicate a claim is made, the more evidence is required. More common sources of errors would seem to be more likely and thus more common causes of bit flips. Thus more evidence is required for the cosmic ray hypothesis being a dominant reason than anything else. We know…
That's not contrasting evidence. Defects are certainly common sources of error, particularly with cheap commodity components like those used in google's fleet. That does not prove cosmic rays aren't a significant source of SEUs [in any computing device].
No but it adds significant credence against the cosmic ray hypothesis. Particularly since this Mozilla blog post has lots of speculation without any actual data to support the claim.
FF is running on even lower cost commodity components without strict controls on the operating environment (eg environmental heat isn’t controlled like it is in data centers, power supplies can be borderline vs carefully species Google machines, etc).
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#86How do they know it's cosmic rays and not something else?
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#87Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#88Earlier quoted context omitted.
I've few years ago hear, US gov't created app, which periodically checked for random light flashes on camera sensor of smartphone, and send info to cloud, to use them as large distributed network for detecting of illegal nukes. (btw, Soviet and Russians really love to collect such weird facts). Idea looks very realistic, except that it will drain battery, and could also be used as surveillance tool, so I have not see…
> "I've few years ago hear, US gov't created app..." At least at one point they had satellites deployed for this purpose. The phenomena they used to determine whether a detonation had taken place was the characteristic 'double flash'. You can read some more about this here: https://en.wikipedia.org/wiki/Vela_incident They don't need to resort to such clandestine measures as infiltrating people's smartphones for this…
> satellites deployed for this purpose
You missing point. I know about world system of monitoring for nuke explosions, but I don't talk about it.
I mean, threat of use nuke by terrorists is very close by its consequences to real blast.
Imagine, how much panic will happen, if appear plausible claim, that terrorists have nuke, which they planning to use in one of biggest US cities.
So special semi-military agencies, constantly working on eliminating possibility that terrorists will have access even to radioactive waste, not even speaking about enriched materials.
And this is what differ Western system of values from Russian or Asian - in Western system, lost of even one person considered unacceptable, but Russians accept sacrifice 20k of soldiers for crazy escapade, which they see in what now happen in Ukraine.
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#89I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…
A server with 64 GB of ECC ram sitting at an altitude of 3.2 km on the Greenland ice sheet is reporting... 0 bit errors (whether correctable or uncorrectable) in the 244 days it's been up. A server with 16 GB of ECC ram at an altitude of 3.8 km in California is reporting.... 0 bit errors in the 146 days it's been up. Maybe I shouldn't believe what /sys/devices/system/edac/mc is reporting? These are EL8 systems...
Bryan Cantrill mentions in this talk that they saw no correctable errors, until they suddenly got uncorrectable errors. Turns out that the firmware was hiding the fact that it had been correcting ECC errors the whole time.
https://youtu.be/vvZA9n3e5pc?t=1193
Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla
#90I know HN has a decent Factorio fanbase. Factorio properly stresses PC hardware, and borderline memory is usually ok for a casual gamer until you start a Factorio megabase. A decent example is Warger who does speedruns: https://forums.factorio.com/viewtopic.php?f=7&t=100646 https://www.speedrun.com/factorio#100 Those that have played the game - speedruns are amazing to watch, if you haven't already.
The FFF about bugfixing talking about potential cosmic ray (or bad hardware) corruption :