Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How NASA built Artemis II’s fault-tolerant computer

cacm.acm.org

Post reply on HN
Print view View original

231–240 of 257 posts

Re: How NASA built Artemis II’s fault-tolerant computer

#231

Post by spwa4 »

znort_ wrote: post #155
spwa4 wrote: post #144

Earlier quoted context omitted.

You should read up on Yanis Varoufakis' history and just how bad his solution for Greece went. That will explain the extreme amounts of anger on both his side, the side of Greeks and the side of the EU and worldwide financial community (and the EU itself used to be an industry cartel, so you can guess how much every government institution in the EU aligns with the worldwide financial community). This guy will never b…

> He caused a MAJOR issue for Greece that still affects everyone in his country today, after reassuring people for 2+ years it was never going to happen: care to explain what exactly he caused and how that still affects everyone in his country? in particular how he managed to jump several years backward in the timeline?

All I can say is "keep reading". Because it takes a BIG turn for the worse at one point, and that's where he's involved.

Re: How NASA built Artemis II’s fault-tolerant computer

#232

Post by [deleted] »

budman1 wrote: post #211
ehnto wrote: post #204

Earlier quoted context omitted.

It still lives on as a bit of a hard skill in automotive/robotics. As someone who crosses the divide between enterprise web software, and hacking about with embedded automotive bits, I don't really lament that we're not using WCET and Real Time OSes in web applications!

ever use wordstar on Z80 system with a 5 MB hard drive? responsive. everything dealing with user interaction is fast. sure, reading a 1 MB document took time, but 'up 4 lines' was bam!. linux ought to be this good, but the I/O subsystem slows down responsiveness. it should be possible to copy a file to a USB drive, and not impact good response from typing, but it is not. real time patches used to improve it. windows…

[deleted]

Re: How NASA built Artemis II’s fault-tolerant computer

#233

Post by coryrc »

wincy wrote: post #227
coryrc wrote: post #222

Earlier quoted context omitted.

Every single hardware subsystem adds lag. Double buffering adds a frame of lag; some do triple-buffering. USB adds ~8ms worse-case. LCD TVs add their own multi-frame lag-inducing processing, but even the ones that don't have to load the entire frame before any of it shows, which can be a substantial fraction of the time between frames. Those old systems were "racing the beam", generating every pixel as it was being d…

8khz polling rate mouse and keyboard, 240hz 4K monitor (with Oled to reduce smearing preferably, or it becomes very noticeable), 360hz 1440p, or 480hz 1080p, is current state of the art. You need a decent processor and GPU (especially the high refresh rate monitors as you’re pushing a huge amount data to your display, as only the newest GPUs support the newest display port standard) to run all this, but my Windows de…

That's a good one. I probably should have brought up variance though. These cache-less systems had none. Windows might just decide to index a bunch of stuff and trash your cache, and it runs slow for a bit while loading gigabytes of crap back into memory. When I flip my lightswitch, it's always (perceptibly) the same amount of time until the light comes on. Click a button on the screen? Uh...

Re: How NASA built Artemis II’s fault-tolerant computer

#234

Post by cpgxiii »

anymouse123456 wrote: post #171
dmk wrote: post #3

The quote from the CMU guy about modern Agile and DevOps approaches challenging architectural discipline is a nice way of saying most of us have completely forgotten how to build deterministic systems. Time-triggered Ethernet with strict frame scheduling feels like it's from a parallel universe compared to how we ship software now.

Some of us still work on embedded systems with real-time guarantees. Believe it or not, at least some of those modern practices (unit testing, CI, etc) do make a big (positive) difference there.

The depressing part is that these "modern practices" were essentially invented in the 1960s by defense and aerospace projects like the NTDS, LLRV/LLTV, and Digital Fly-by-Wire to produce safety-critical software, and the rest of the software industry simply ignored them until the last couple of decades.

Re: How NASA built Artemis II’s fault-tolerant computer

#235

Post by neuralRiot »

iririririr wrote: post #221
ehnto wrote: post #204

Earlier quoted context omitted.

It still lives on as a bit of a hard skill in automotive/robotics. As someone who crosses the divide between enterprise web software, and hacking about with embedded automotive bits, I don't really lament that we're not using WCET and Real Time OSes in web applications!

if you ever worked on automotive you know it's bs. since CAN all reliability and predictive nature was out. we now have redundancy everywhere with everything just rebooting all the time. install an aftermarket radio and your ecu will probably reboot every time you press play or something. and that's just "normal".

I’ve working in automotive since it was only wires and never saw that (or noticed it) happening specially since usually body and powertrain work on separate buses tied through a gateway, the crazy stuff happens when people start treating the bus (specially the higher speed ones) like a 12v line or worst.

Re: How NASA built Artemis II’s fault-tolerant computer

#236

Post by lrvick »

ranger207 wrote: post #207
lrvick wrote: post #143

NASA describes some impressive work for runtime integrity, but the lack of mention of build-time security is surprising. I would expect to see multi-party-signed deterministic builds etc. Anyone have any insight here?

What would the threat profile be here to require that? Regardless, I'd be a little surprised if they didn't have anything like that; provenance is very important in aerospace, with hardware tracked to the point that NTSB investigators looking at a crash can tell what ingot a bolt was made from

In my experience government just uses RedHat which is -not- reproducible and -not- full source bootstrapped so a single person in the supply chain could maliciously or accidentally backdoor everything. Maybe the goal of the supply chain attacker is just embarrassing the Americans at best or cause a material loss of life at worst.

I would -hope- NASA does not trust their OS supply chains to a single person for high risk applications, but given even major companies I audit do this with billions of dollars on the line, it would not shock me if NASA has the same stance which worries me a bit.

They would need to be using something like heavily customized buildroot or stagex to produce deterministic OS images.

Re: How NASA built Artemis II’s fault-tolerant computer

#237

Post by weli »

nnevod wrote: post #223
weli wrote: post #135

Earlier quoted context omitted.

I don't know if you are being sarcastic. But no, it's not an "utopia" by any means and the free market still has many pitfalls and problems that I described. However, is the best system we have to coordinate the production, distribution and purchasing of services and goods on a mass scale.

Somehow the last sentence of your comment caught me as if there's something wrong with it. I don't thing it's wrong, but I think it should be generalised. Free market is an approach to negotiation, analogous to ad-hoc model in computer science, as opposed to client-server model - which matches command economy. There are tons of nuances of course regarding incentives, optimal group sizes, resource constraints etc. Fre…

> Also there's the, often, I suppose, intentional confusion of terms. The free market of the economic theory is not an unregulated market, it's a market of infinitesimal agents with infinitesimal influence of each individual agent upon the whole market, with no out-of-market mechanisms and not even in-market interaction between agents on the same side.

Just to expand on this really interesting topic. That's where the common pitfall on planned economy begins. Because to some degree a free market can withstand some amount of regulation; after all, external agents trying to manipulate the market are just that, agents in the market. As long as there are other autonomous agents intervening the market will keep functioning as it was. So the bureaucrat has both the incentive and the justification to expand the intervention. In other words, his economical plan didn't work because it was not intervened enough and just if they intervene in this extra thing it will work for sure. That loop continues until the market is 100% intervened, and at that point it requires such a enormous structure of power and control that makes it difficult to fight it (clientelist networks, repressive states, etc).

Re: How NASA built Artemis II’s fault-tolerant computer

#238

Post by HeyLaughingBoy »

MikeTheGreat wrote: post #84
vasco wrote: post #72

Earlier quoted context omitted.

If you have a point to make, make it.

What my question is hinting at is that there's actually some really interesting engineering around resolving what happens when the systems disagree. Things like Paxos and Raft help make this much more tractable for mere mortals (like myself); the logic and reasoning behind them are cool and interesting.

There really is. We designed a redundant system (software, hardware and mechanisms) a couple years ago. And the problems around figuring out who's in control and how to keep things synchronized across a number of potential failure modes gets really hairy. Sadly, the project was cancelled before we could complete the implementation.

Re: How NASA built Artemis II’s fault-tolerant computer

#239

Post by sheepybloke »

anthonj wrote: post #91
__d wrote: post #34

Does anyone have pointers to some real information about this system? CPUs, RAM, storage, the networking, what OS, what language used for the software, etc etc? I’d love to know how often one of the FCMs has “failed silent”, and where they were in the route and so on too, but it’s probably a little soon for that.

Nasa CFS, is written is plain C (trying to follow MISRA C, etc). It's open on girhub abd used by many companies. It's typically run over freertos or RTEMS, not sure here. Personally I find the project extremely messy, and kinda hate working with it.

It's most likely using vxworks for it's OS, since I believe it's one of the only fully certified ARINC653 OS's for human flight. It's used in most Aircraft and space missions.

Re: How NASA built Artemis II’s fault-tolerant computer

#240

Post by sillywalk »

dojopico wrote: post #173

I did VOS and database performance stuff at Stratus from 1989-95. Stratus was the hardware fault tolerant company. Tandem, our arch rivals, did software fault tolerance. Our architecture was “pair and spare”. Each board had redundant everything and was paired with a second board. Every pin out was compared on every tick. Boards that could not reset called home. The switch from Motorola 68K to Intel was a nightmare fo…

> Stratus was the hardware fault tolerant company. Tandem, our arch rivals, did software fault tolerance. Our architecture was “pair and spare”.

To expand on this, when Tandem switched MIPS from their proprietary processors, the CPUs were duplicated on a board and compared, and if they disagreed, the logical CPU would halt, similar to Stratus. The software-pair backup processes in a different logical CPU would then take over.

Post reply on HN