How NASA built Artemis II’s fault-tolerant computer • Hacker News Board

Re: How NASA built Artemis II’s fault-tolerant computer

#241

Post by KennyBlanken » Sat, Apr 11, 2026, 4:17 AM UTC

JumpCrisscross wrote: post #102

themafia wrote: post #61

Earlier quoted context omitted.

In the Shuttle they would use command averaging. All four computers would get access to an actuator which would tie into a manifold which delivered power to the flight control surface. If one disagreed then you'd get 25% less command authority to that element.

> In the Shuttle they would use command averaging I think the Shuttle, operating only in LEO, had more margin for error. Averaging a deep-space burn calculation is basically the same as killing the crew.

Fucking up the re-entry burn or thruster actuation during the burn for re-entry = loss of vehicle/crew

Improper control surface actuation during re-entry = loss of vehicle/crew

Also, rocket engines that are powered by the combustion of their fuel and oxidizer (the exhaust gasses of which drive the main pumps) have a very specific startup sequence. For example, if any of the combustion chambers have a mix of oxygen and hydrogen too close to stochiometric when the igniters fire, you get an explosion, not a burn. Not too dissimilar from what happens in car engines when you get detonation (which is very different from knocking. Detonation melts holes in stuff.)

Startup initially is open-loop with no feedback or adjustment based on sensors and then at some point the computer switches over to closed loop control. It starts with hydrogen first. The sparklers? Those aren't for igniting the engine, that's done by igniters inside the combustion chamber(s). The sparklers are to ignite all the hydrogen that is pushed out the nozzle initially so there's a very fuel-rich environment in the engine and it doesn't go kaboom.

If things go wrong - such as a valve not opening as fast as it should, or not being opened the right amount at the right time - the engine goes kaboom. This happened to a bunch of engines during development and testing.

But Artemis has basically the same engines, so...shrug

Re: How NASA built Artemis II’s fault-tolerant computer

#242

Post by MathMonkeyMan » Sat, Apr 11, 2026, 5:46 AM UTC

geomark wrote: post #55

I sure wish they would talk about the hardware. I spent a few years developing a radiation hardened fault tolerant computer back in the day. Adding redundancy at multiple levels was the usual solution. But there is another clever check on transient errors during process execution that we implemented that didn't involve any redundancy. Doesn't seem like they did anything like that. But can't tell since they don't ment…

I read, for probe missions, that one technique is to get a bunch of consumer chips and irradiate the hell out of them. Now take the winner model and get a bunch of those. Irradiate them. The winner goes to Mars.

The claim was that some plain old chips are exquisitely radiation resisitant, and it's not clear why.

Re: How NASA built Artemis II’s fault-tolerant computer

#243

Post by rurban » Sat, Apr 11, 2026, 7:49 AM UTC

Raft consensus with pairs? I smell bulls*t there. Even when they say it's 8, it boils down to pair-wise checks, without any consensus. Just the consensus of wrong.

Also https://en.wikipedia.org/wiki/TTEthernet looks like bolting time-guaranteed switching networks onto randomizing ethernet hardware. Sounds incredibly cheap and stupid. Either stay with guaranteed real-time switching, or give up on hard real-time guarantees and favor performance, simplicity and cheap stock hardware.

Monkeys in space.

Re: How NASA built Artemis II’s fault-tolerant computer

#244

Post by MutexMaven » Sat, Apr 11, 2026, 1:53 PM UTC

geomark wrote: post #55

I sure wish they would talk about the hardware. I spent a few years developing a radiation hardened fault tolerant computer back in the day. Adding redundancy at multiple levels was the usual solution. But there is another clever check on transient errors during process execution that we implemented that didn't involve any redundancy. Doesn't seem like they did anything like that. But can't tell since they don't ment…

NASA actually publishes these things on their NTRS page. The Primary flight controller is rocking Green Hills INTEGRITY RTOS on BAE RAD750s in a quad redundant config, with a VxWorks backup on a Frontgrade Gaisler LEON4 (SPARC V8). This allowed for parts of the ARINC653 spec regarding time and space partitioning of the RTOS scheduler to be used.

You can read more about it below (when the server throwing errors). https://ntrs.nasa.gov/api/citations/20190000011/downloads/20... https://ntrs.nasa.gov/api/citations/20230002185/downloads/FS...

Re: How NASA built Artemis II’s fault-tolerant computer

#245

Post by MutexMaven » Sat, Apr 11, 2026, 1:58 PM UTC

Schlagbohrer wrote: post #180

"High-performance supercomputers are used for large-scale fault injection, emulating entire flight timelines where catastrophic hardware failures are introduced to see if the software can successfully ‘fail silent’ and recover." I assume this means they are using a digital twin simulation inside the HPC?

Yes, they leveraged Intel Simics and many other tools like Matlab etc. to have "Digital Twin" simulations.

The extensive use of simulators and emulators has been particularly critical, enabling parallel design and development workflows to compensate for the incredibly expensive and long-lead times of hardware. So this helped with bottlenecks in development too.

https://ntrs.nasa.gov/api/citations/20190000011/downloads/20...

Re: How NASA built Artemis II’s fault-tolerant computer

#246

Post by MutexMaven » Sat, Apr 11, 2026, 2:12 PM UTC

MutexMaven wrote: post #245

Schlagbohrer wrote: post #180

"High-performance supercomputers are used for large-scale fault injection, emulating entire flight timelines where catastrophic hardware failures are introduced to see if the software can successfully ‘fail silent’ and recover." I assume this means they are using a digital twin simulation inside the HPC?

Yes, they leveraged Intel Simics and many other tools like Matlab etc. to have "Digital Twin" simulations. The extensive use of simulators and emulators has been particularly critical, enabling parallel design and development workflows to compensate for the incredibly expensive and long-lead times of hardware. So this helped with bottlenecks in development too. https://ntrs.nasa.gov/api/citations/20190000011/download…

Wrong link, but still relevant. https://ntrs.nasa.gov/api/citations/20140009920/downloads/20...

Re: How NASA built Artemis II’s fault-tolerant computer

#247

Post by programmertote » Sat, Apr 11, 2026, 5:18 PM UTC

albertzeyer wrote: post #188

I'm curious: In the current moon flyby, how often did some of these fallback methods get active? Was the BFS ever in control at any point? How many bitflips were there during the flight so far?

The same question I wanted to ask. I'd be very curious to learn about their post-mission analysis to find out how many bit flips occurred and how many times this redundant system prevented the mistakes from causing issues.

Re: How NASA built Artemis II’s fault-tolerant computer

#248

Post by estimator7292 » Sat, Apr 11, 2026, 9:57 PM UTC

rurban wrote: post #243

Raft consensus with pairs? I smell bulls*t there. Even when they say it's 8, it boils down to pair-wise checks, without any consensus. Just the consensus of wrong. Also https://en.wikipedia.org/wiki/TTEthernet looks like bolting time-guaranteed switching networks onto randomizing ethernet hardware. Sounds incredibly cheap and stupid. Either stay with guaranteed real-time switching, or give up on hard real-time guaran…

Extremely bold take to insist you're smarter than people who literally just flew to the actual moon and back.

Re: How NASA built Artemis II’s fault-tolerant computer

#249

Post by __d » Sun, Apr 12, 2026, 1:43 AM UTC

sheepybloke wrote: post #239

anthonj wrote: post #91

Earlier quoted context omitted.

Nasa CFS, is written is plain C (trying to follow MISRA C, etc). It's open on girhub abd used by many companies. It's typically run over freertos or RTEMS, not sure here. Personally I find the project extremely messy, and kinda hate working with it.

It's most likely using vxworks for it's OS, since I believe it's one of the only fully certified ARINC653 OS's for human flight. It's used in most Aircraft and space missions.

Yeah, that was my guess too but the comment about separate implementation for the backup system made me wonder if there was a different OS, and the which was running where.

Re: How NASA built Artemis II’s fault-tolerant computer

#250

Post by __d » Sun, Apr 12, 2026, 1:49 AM UTC

j4k0bfr wrote: post #145

__d wrote: post #34

Does anyone have pointers to some real information about this system? CPUs, RAM, storage, the networking, what OS, what language used for the software, etc etc? I’d love to know how often one of the FCMs has “failed silent”, and where they were in the route and so on too, but it’s probably a little soon for that.

Not sure about the primary FSW but the BFS uses cFS[0]. As the sibling comment mentions, you can check it out on GitHub. Sadly I believe NASA keeps most of their best code private, probably siloed into mission-specific codebases. Still, the cFS repo is an awesome crash course on old-school Flight Software techniques. [0] https://youtu.be/4doI2iQe4Jk?si=ucMoIdw7x_QgZR32

Helpful video, thanks!

At about 1:20, the presenter says the BFS uses a different OS and hardware (not sure if that means a different instance, or a different class, so to speak).