Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airbus A320 – intense solar radiation may corrupt data critical for flight

airbus.com

Post reply on HN
Print view View original

171–180 of 184 posts

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#171

Post by exidy »

themerone wrote: post #26
jayanmn wrote: post #15

Earlier quoted context omitted.

I am worried about a software fix for what looks like hardware problem.

Gracefully handling hardware faults is a software problem. The Air France Flight 447 crash was the result of bad software and bad hardware.

Although the pitot tubes on AF447 were due to be replaced with a type more resistant to icing, nonetheless there's no such thing as a 100% reliable pitot tube and there were procedures to follow in the event of unreliable airspeed indication. Had they been followed the accident would not have happened. Instead the co-pilot held back on his stick until the aircraft fell out of the sky.

I don't believe there was any issue identified with the software of the plane.

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#172

Post by asdefghyk »

RealityVoid wrote: post #46
addaon wrote: post #7

I’d really, really like to know what microcontroller family this was found on. Assuming that this is a safety processor (lockstep, ECC, etc) it suggests that ECC was insufficient for the level of bit flips they’re seeing — and if the concern is data corruption, not unintended restart, it means it’s enough flips in one word to be undetectable. The environment they’re operating in isn’t that different from everyone els…

See my other comments in the other threads. This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. That flight computer was designed in the 90's and updated in 2002 with a new hw variant that does have edac. So yes, for this kind of thing, I can buy that a bit flip happened. You can see much more data in the report: https://www.atsb.gov.au/sit…

Specifically of the above document , APPENDIX G: ELECTROMAGNETIC RADIATION mentions in detail how radiation (possible from the sun ) can cause flipped bits and other errors in electronic circuits in some detail ..... We are also at the PEAK of the 11 year sunspot...

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#173

Post by asdefghyk »

addaon wrote: post #7

I’d really, really like to know what microcontroller family this was found on. Assuming that this is a safety processor (lockstep, ECC, etc) it suggests that ECC was insufficient for the level of bit flips they’re seeing — and if the concern is data corruption, not unintended restart, it means it’s enough flips in one word to be undetectable. The environment they’re operating in isn’t that different from everyone els…

RE "...The environment they’re operating in isn’t that different from everyone else...." NO this is incorrect. High flying aircraft more likely to suffer increased radiation caused by 11 year peak sunspot cycle . such aircraft should be using "radiation hardened electronics" , somewhat like spacecraft use...

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#174

Post by asdefghyk »

Intense solar radiation will be at a peak, since it is NOW the peak of the 11 year sunspot cycle.

Good related reading on this page .... https://en.wikipedia.org/wiki/Radiation_hardening ... includes a range of mitigation effects.

( I would be interested to find out how they actually test these systems. What combinations of hardware hardening and software logic. ALso do they actually subject to system to radiation as part of the testing )

https://en.wikipedia.org/wiki/Radiation_hardening

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#175

Post by lxgr »

probably_wrong wrote: post #110
kijin wrote: post #42

Earlier quoted context omitted.

I imagine it could help with Airbus marketing. "We take proactive measures, whereas our competitor only takes action after multiple fatal crashes!"

I know someone who is stranded in another continent thanks to this. Trust me, all the understanding I could have as a technical user has been offset by the MASSIVE pain in the ass that is rebooking an international flight. And non-technical users have heard "the plane will not travel because it requires a software update", which does not inspire confidence. As far as I'm concerned it has not helped with their marketi…

> "the plane will not travel because it requires a software update", which does not inspire confidence.

It actually inspires a lot of confidence to people who can at least think economically, if not technically:

Grounding thousands of planes is very expensive (passengers get cash for that in at least the EU, and sometimes more than the ticket cost!), so doing it both shows that it’s probably a serious issue and it’s being taken seriously.

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#176

Post by probably_wrong »

lxgr wrote: post #175
probably_wrong wrote: post #110

Earlier quoted context omitted.

I know someone who is stranded in another continent thanks to this. Trust me, all the understanding I could have as a technical user has been offset by the MASSIVE pain in the ass that is rebooking an international flight. And non-technical users have heard "the plane will not travel because it requires a software update", which does not inspire confidence. As far as I'm concerned it has not helped with their marketi…

> "the plane will not travel because it requires a software update", which does not inspire confidence. It actually inspires a lot of confidence to people who can at least think economically, if not technically: Grounding thousands of planes is very expensive (passengers get cash for that in at least the EU, and sometimes more than the ticket cost!), so doing it both shows that it’s probably a serious issue and it’s…

First, I feel the implication that "if you aren't reassured is only because you're dumb" is unwarranted.

With that out of the way, being expensive does not preclude shoddy work. At the end of the day, the only difference between "they are so concerned about security that they are willing to lose millions[1]" and "their process must be so bad that they have no other choice but to lose millions before their death trap cost them ten times that" is how good your previous perception of their airplanes is.

I think that, had this exact same issue happened to Boeing, we would be having a very different conversation. As the current top-comment suggests, it would probably be less "these things happen" and more "they cheapened out on the ECC".

[1] Disclaimer: I have no idea who loses money in this scenario, if it's also Airbus or if it's exclusively the airlines who bought them.

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#177

Post by p_l »

lxgr wrote: post #80
RealityVoid wrote: post #46

Earlier quoted context omitted.

See my other comments in the other threads. This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. That flight computer was designed in the 90's and updated in 2002 with a new hw variant that does have edac. So yes, for this kind of thing, I can buy that a bit flip happened. You can see much more data in the report: https://www.atsb.gov.au/sit…

> This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. Wasn't the philosophy back then to run multiple independent (and often even designed and manufactured by different teams) computers and run a quorum algorithm at a very high level? Maybe ECC was seen as redundant in that model?

One thing to remember is that unless you explicitly made a device to operate in space and even there beyond LEO, you quite probably might have seen no requirement for ECC memory for radiation reasons.

ECC memory usage in the past was heavily correlated with, well, way lower quality of the hardware from chips to assembly, electromagnetic interference from unexpected sources, or even customer/field technician errors. Remember an early 1980s single user workstation might require extensive check & fix cycle just from moving it around.

An aircraft component would eliminate all major parts of that, including both through more thorough self-testing, careful sealed design, selection of high grade parts, etc.

The possibility of space radiation causing considerable issues came up as fully digital fly by wire became more common in civilian usage and has led over time to retrofitting with EDAC, but radiation-triggered SEU was deemed low enough risk due to design of the system.

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#178

Post by addaon »

p_l wrote: post #177
lxgr wrote: post #80

Earlier quoted context omitted.

> This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. Wasn't the philosophy back then to run multiple independent (and often even designed and manufactured by different teams) computers and run a quorum algorithm at a very high level? Maybe ECC was seen as redundant in that model?

One thing to remember is that unless you explicitly made a device to operate in space and even there beyond LEO , you quite probably might have seen no requirement for ECC memory for radiation reasons. ECC memory usage in the past was heavily correlated with, well, way lower quality of the hardware from chips to assembly, electromagnetic interference from unexpected sources, or even customer/field technician errors.…

> One thing to remember is that unless you explicitly made a device to operate in space and even there beyond LEO, you quite probably might have seen no requirement for ECC memory for radiation reasons.

This does not match my experience (although, admittedly, I've been in the field only a couple decades -- the hardware under discussion predates that). The problem with SEU-induced bit flips is not that errors happen, but that errors with unbounded behavior happen -- consider a bit flip in the program counter, especially in an architecture with variable sized instructions. This drives requirements around error detection, not correction -- but the three main tools here are lockstep processor cores, parity on small memories, and SECDED on large memories. SECDED ECC here is important both because it can catch double errors that happen close together in time, and because memory scrubbing with single error correction allows multiple errors spaced in time to be considered separately. At the system level, the key insight is that detectable failures of a single ECU have to be handled anyway, because of non-transient statistical failures -- connector failures, tin whiskers, etc. The goal, then, is to convert substantially all failures to detectable failures, and then have defined failure behavior (often fail-silent). This leads to dual-dual redundancy architectures and similar, instead of triplex; each channel consists of two units that cross-check each other, and downstream units can assume that commands received from either channel are either correct or absent.

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#179

Post by p_l »

addaon wrote: post #178
p_l wrote: post #177

Earlier quoted context omitted.

One thing to remember is that unless you explicitly made a device to operate in space and even there beyond LEO , you quite probably might have seen no requirement for ECC memory for radiation reasons. ECC memory usage in the past was heavily correlated with, well, way lower quality of the hardware from chips to assembly, electromagnetic interference from unexpected sources, or even customer/field technician errors.…

> One thing to remember is that unless you explicitly made a device to operate in space and even there beyond LEO, you quite probably might have seen no requirement for ECC memory for radiation reasons. This does not match my experience (although, admittedly, I've been in the field only a couple decades -- the hardware under discussion predates that). The problem with SEU-induced bit flips is not that errors happen,…

The incident report on the 2008 case specifically mentions SEU for memory - certain level of EDAC techniques is applied on the whole unit, but one area that was not covered was the possibility of non-catastrophic (in terms of operation) failure of memory module with a bitflip.

An under-appreciated thing is also that the devices in question used to be rebooted pretty often which triggered self-test routines in addition to the run-time tests - something that didn't trigger anything in case of A330 in 2008, but was impactful in risk assessments missing certain things with 787 some years later (and newer A380/A350 recently).

Re: Airbus A320 – intense solar radiation may corrupt data critical for flight

#180

Post by p_l »

Reason077 wrote: post #56
RealityVoid wrote: post #46

Earlier quoted context omitted.

See my other comments in the other threads. This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. That flight computer was designed in the 90's and updated in 2002 with a new hw variant that does have edac. So yes, for this kind of thing, I can buy that a bit flip happened. You can see much more data in the report: https://www.atsb.gov.au/sit…

The recalled aircraft include the latest A320neo model, some of which are basically brand new. Why would they be using flight computers from before 2002? Why is an old report from 2008, relating to a completely different aircraft type (A330), relevant to the A320 issue today?

For the same reasons that Honeywell is building new devices with AMD 29050 CPUs today[1] - by sticking with the same CPU ISA, they can avoid recertifying portion of the software stack.

[1] Honeywell actually bought full license et al from AMD and operates a fabless team that ensures they have stock and if necessary updates the chip.

Post reply on HN