Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Flips Your Bit: Cosmic Ray Errors at Mozilla

blog.mozilla.org

Post reply on HN
Print view View original

21–30 of 96 posts

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#21

Post by bqmjjx0kac »

jldugger wrote: post #12
robotsteve2 wrote: post #7

Any sort of hardware or software error seems much more likely. Computers are incredibly complex and approximations are used everywhere (in the design of the hardware, in the theory of operation). I don't think inference-based experiments or analysis on cosmic ray bit flips are appropriate. You really need some kind of dedicated cosmic ray detector nearby as a control. If the flux of cosmic rays into the detector is o…

Indeed, there was a study in IEEE pointing out the absurdity of cosmic rays as causes -- one point cited was that the vast majority of bitflip happen at specific points in the address space, page boundaries between chips essentially

I'm curious why that is evidence against the cosmic ray explanation.

Couldn't it have something to do with the physical layout of memory? Perhaps those page-boundary-adjacent addresses present a larger physical target, perhaps on the bus.

Of course I am wildly speculating right now. I'd love to see the article if you have a link!

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#22

Post by rat9988 »

xenophonf wrote: post #20
pitaj wrote: post #19

Earlier quoted context omitted.

Is one mitigation TLS certificate verification?

Quoted post unavailable.

> Why not read the linked paper,

You answered your own question. The answer is in page 12, which means there is too much information. He is not interested in the whole topic, just about this question. So he asks, maybe someone is charitable enough to answer. Nothing wrong with it.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#23

Post by geophile »

As the article points out, using collected client data is problematic, because some errors will often be undetectable, as in numeric data. And in general, you would have to control for bit flips somehow caused by software.

I wonder whether a SETI approach would be useful here. Allocate, say, 1MB of memory. Fill it with some known bit pattern. Periodically check the memory and look for discrepancies. Do this once an hour, on 10M devices, and that is a LOT of monitoring. Report discrepancies along with time, location (including elevation), hardware and OS information.

I would think that this approach would provide a lot of interesting information about when and where bit flips occur, especially when matched against information on solar and atmospheric events (as in the article). Perhaps sensitive hardware and OS environments would be detected. Even completely negative results would be interesting: no bit flips observed would suggest that purported bit flips elsewhere might have other explanations.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#24

Post by tedunangst »

rat9988 wrote: post #22
xenophonf wrote: post #20

Earlier quoted context omitted.

Quoted post unavailable.

> Why not read the linked paper, You answered your own question. The answer is in page 12, which means there is too much information. He is not interested in the whole topic, just about this question. So he asks, maybe someone is charitable enough to answer. Nothing wrong with it.

No post body was provided.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#25

Post by axg11 »

antognini wrote: post #6
axg11 wrote: post #4

This is fascinating and hints at a future possible scientific study: using phones across the globe to map cosmic ray events. I'm not a physicist so I can't speak for the value of such data. If cosmic ray events do not occur uniformly across the globe then mapping events from 100,000s of phones could give interesting insights.

As they say, there's an app for that: https://cosmicrayapp.com/ Basically it monitors your camera for the streaks produced by cosmic rays. You can see the real time stream of events here: https://cosmicrayobserver.com/#0.4/0/0

This is amazing - thank you for sharing!

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#27

Post by xenophonf »

rat9988 wrote: post #22
xenophonf wrote: post #20

Earlier quoted context omitted.

Quoted post unavailable.

> Why not read the linked paper, You answered your own question. The answer is in page 12, which means there is too much information. He is not interested in the whole topic, just about this question. So he asks, maybe someone is charitable enough to answer. Nothing wrong with it.

How could I possibly answer their question better than the experts who wrote the paper?

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#28

Post by jeffreygoesto »

incomingpain wrote: post #11

I had the opportunity to design my SOC from scratch. Mostly ripping off Berkeley's public design. Something I have documented in the last 2 years. Solar flare activity is what causes problems. All memory is ECC but it still happens. Faraday cage incoming? Wait? Faraday cage racks million $ idea?

Using an FD-SOI process can help reducing soft errors.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#29

Post by rat9988 »

xenophonf wrote: post #27
rat9988 wrote: post #22

Earlier quoted context omitted.

> Why not read the linked paper, You answered your own question. The answer is in page 12, which means there is too much information. He is not interested in the whole topic, just about this question. So he asks, maybe someone is charitable enough to answer. Nothing wrong with it.

How could I possibly answer their question better than the experts who wrote the paper?

Maybe you can't but someone else can. The question is open to anyone who can and wants to answer.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#30

Post by EscargotCult »

geophile wrote: post #23

As the article points out, using collected client data is problematic, because some errors will often be undetectable, as in numeric data. And in general, you would have to control for bit flips somehow caused by software. I wonder whether a SETI approach would be useful here. Allocate, say, 1MB of memory. Fill it with some known bit pattern. Periodically check the memory and look for discrepancies. Do this once an h…

This could be crowdsourced. Imagine some sort of reporting network of volunteers, running the simple program (memory allocation and periodic checks) on any hardware they're loaning time on, and submitting their location and altitude as well.
Post reply on HN