Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How NASA built Artemis II’s fault-tolerant computer

cacm.acm.org

Post reply on HN
Print view View original

221–230 of 257 posts

Re: How NASA built Artemis II’s fault-tolerant computer

#221

Post by iririririr »

ehnto wrote: post #204
carefree-bob wrote: post #66

Earlier quoted context omitted.

During the time of the first Apollo missions, a dominant portion of computing research was funded by the defense department and related arms of government, making this type of deterministic and WCET (worst case execution time) a dominant computing paradigm. Now that we have a huge free market for things like online shopping and social media, this is a bit of a neglected field and suffers from poor investment and mind…

It still lives on as a bit of a hard skill in automotive/robotics. As someone who crosses the divide between enterprise web software, and hacking about with embedded automotive bits, I don't really lament that we're not using WCET and Real Time OSes in web applications!

if you ever worked on automotive you know it's bs.

since CAN all reliability and predictive nature was out. we now have redundancy everywhere with everything just rebooting all the time.

install an aftermarket radio and your ecu will probably reboot every time you press play or something. and that's just "normal".

Re: How NASA built Artemis II’s fault-tolerant computer

#222

Post by coryrc »

budman1 wrote: post #211
ehnto wrote: post #204

Earlier quoted context omitted.

It still lives on as a bit of a hard skill in automotive/robotics. As someone who crosses the divide between enterprise web software, and hacking about with embedded automotive bits, I don't really lament that we're not using WCET and Real Time OSes in web applications!

ever use wordstar on Z80 system with a 5 MB hard drive? responsive. everything dealing with user interaction is fast. sure, reading a 1 MB document took time, but 'up 4 lines' was bam!. linux ought to be this good, but the I/O subsystem slows down responsiveness. it should be possible to copy a file to a USB drive, and not impact good response from typing, but it is not. real time patches used to improve it. windows…

Every single hardware subsystem adds lag. Double buffering adds a frame of lag; some do triple-buffering. USB adds ~8ms worse-case. LCD TVs add their own multi-frame lag-inducing processing, but even the ones that don't have to load the entire frame before any of it shows, which can be a substantial fraction of the time between frames.

Those old systems were "racing the beam", generating every pixel as it was being displayed. Minimum lag was microseconds. With LCDs you can't get under milliseconds. Luckily human visual perception isn't /that/ great so single-digit milliseconds could be as instantaneous, if you run at 100 Hz without double-buffering (is that even possible anymore!?) and use a low-latency keyboard (IIRC you can schedule more frequent USB frames at higher speeds) and only debounce on key release.

Re: How NASA built Artemis II’s fault-tolerant computer

#223

Post by nnevod »

weli wrote: post #135
mcdeltat wrote: post #132

Earlier quoted context omitted.

> optimal Nash equilibrium where both offer and demand are balanced This roughly translates to "optimal utopian society which cannot be criticised in any way" right? Right??

I don't know if you are being sarcastic. But no, it's not an "utopia" by any means and the free market still has many pitfalls and problems that I described. However, is the best system we have to coordinate the production, distribution and purchasing of services and goods on a mass scale.

Somehow the last sentence of your comment caught me as if there's something wrong with it. I don't thing it's wrong, but I think it should be generalised.

Free market is an approach to negotiation, analogous to ad-hoc model in computer science, as opposed to client-server model - which matches command economy. There are tons of nuances of course regarding incentives, optimal group sizes, resource constraints etc.

Free market is also like evolution - it creates thing that work. Not perfect, not best, they just work. Fitting the situation, not much else (there is always a random chance of something else).

Also there's the, often, I suppose, intentional confusion of terms. The free market of the economic theory is not an unregulated market, it's a market of infinitesimal agents with infinitesimal influence of each individual agent upon the whole market, with no out-of-market mechanisms and not even in-market interaction between agents on the same side.

As a side note, I find it sadly amusing that this reasonable discussion is only possible because it's offtopic to the thread's topic. Had the topic been more attractive to more politically and economically agitated folk, the discussion would be more radicalised, I suppose.

Re: How NASA built Artemis II’s fault-tolerant computer

#224

Post by pbronez »

The Artemis computer handles way more flight functions than Apollo did. What are the practical benefits of that?

This electrify & integrate playbook has brought benefits to many industries, usually where better coordination unlocks efficiencies. Sometimes the smarts just add new failure modes and predatory vendor relationships. It’s showing up in space as more modular spacecraft, lower costs and more mission flexibility. But how is this playing out in manned space craft?

Re: How NASA built Artemis II’s fault-tolerant computer

#225

Post by simoncion »

misiek08 wrote: post #90
simoncion wrote: post #8

Earlier quoted context omitted.

> ...they talk as if they have cured cancer. I'd chalk that up to the author of the article writing for a relatively nontechnical audience and asking for quotes at that level.

So the quote is right somewhat, right? If you are writing to non technical people and you use such high wording.

No, it's not right. When put in context, the quote claims that that manner of speaking is used because the speaker has an unwarranted belief that they've done something absolutely incredible and unprecedented. In actuality, the manner of speaking is being used because the intended audience of the article is likely to have little-to-no knowledge of the technical details of what the speaker is talking about.

For example, if the article was aimed at folks who were familiar with the underlying techniques, the last two paragraphs of the "Enforcing Determinism" section would be compressed into [0]

  Each FCM is time-synced and runs a realtime OS. Failures to meet processing deadlines (or excessive clock drift) reset the FCM. Each FCM uses triply-redundant RAM and NICs. *All* components use ECC RAM. Any failures of these components reset the FCM or other affected component.
But you can't assume that a fairly nontechnical audience will understand all that, so your explanation grows long because of all of the basic information it contains. People looking for an excuse to sneer at something will often misinterpret this as the speaker failing to recognize that the basic information they're providing is about things that are basic.

[0] I'm assuming that the time being wildly out of sync will indicate FCM failure and trigger a reset. [1] I'm also assuming that a sufficiently-large failure of a network switch results in the reset of that network switch. If the article was intended for a more technical audience, that level of detail might have been included, but it wasn't, so it isn't.

[1] If it didn't, why even bother syncing the time? I find it a little hard to believe that the FCMs care about anything other than elapsed time, so all you care about is if they're all ticking at the same rate. I expect the way you detect this is by checking for time sync across the FCMs, correcting minor drift, and resetting FCMs with major drift.

Re: How NASA built Artemis II’s fault-tolerant computer

#226

Post by eggy »

Some related good books I have been studying the past few years or so. The Spark book is written by people who've worked on Cube sats:

  * Logical Foundations of Cyber-Physical Systems

  * Building High Integrity Applications with SPARK 

  * Analysable Real-Time Systems: Programmed in Ada

  * Control Systems Safety Evaluation and Reliability (William M. Goble)
I am developing a high-integrity controls system for a prototype hoist to be certified for overhead hoisting with the highest safety standards and targeting aerospace, construction, entertainment, and defense.

Re: How NASA built Artemis II’s fault-tolerant computer

#227

Post by wincy »

coryrc wrote: post #222
budman1 wrote: post #211

Earlier quoted context omitted.

ever use wordstar on Z80 system with a 5 MB hard drive? responsive. everything dealing with user interaction is fast. sure, reading a 1 MB document took time, but 'up 4 lines' was bam!. linux ought to be this good, but the I/O subsystem slows down responsiveness. it should be possible to copy a file to a USB drive, and not impact good response from typing, but it is not. real time patches used to improve it. windows…

Every single hardware subsystem adds lag. Double buffering adds a frame of lag; some do triple-buffering. USB adds ~8ms worse-case. LCD TVs add their own multi-frame lag-inducing processing, but even the ones that don't have to load the entire frame before any of it shows, which can be a substantial fraction of the time between frames. Those old systems were "racing the beam", generating every pixel as it was being d…

8khz polling rate mouse and keyboard, 240hz 4K monitor (with Oled to reduce smearing preferably, or it becomes very noticeable), 360hz 1440p, or 480hz 1080p, is current state of the art. You need a decent processor and GPU (especially the high refresh rate monitors as you’re pushing a huge amount data to your display, as only the newest GPUs support the newest display port standard) to run all this, but my Windows desktop is a joy to use because of all of this. Everything is super snappy. Alternatively, buying an iPad Pro is another excellent way to get very low latencies out of the box.

I really love this blog post from Dan Luu about latency. https://danluu.com/input-lag/

Re: How NASA built Artemis II’s fault-tolerant computer

#229

Post by mrheosuper »

budman1 wrote: post #211
ehnto wrote: post #204

Earlier quoted context omitted.

It still lives on as a bit of a hard skill in automotive/robotics. As someone who crosses the divide between enterprise web software, and hacking about with embedded automotive bits, I don't really lament that we're not using WCET and Real Time OSes in web applications!

ever use wordstar on Z80 system with a 5 MB hard drive? responsive. everything dealing with user interaction is fast. sure, reading a 1 MB document took time, but 'up 4 lines' was bam!. linux ought to be this good, but the I/O subsystem slows down responsiveness. it should be possible to copy a file to a USB drive, and not impact good response from typing, but it is not. real time patches used to improve it. windows…

I believe this is kind of survivor-bias. It's very rare that RTOSes have to handle allocating GBs of data, or creating thousands of processes. I think if current RTOSes run the same application, there would be no noticeable difference compared to mainstream OS(Could be even worse because the OS is not designed for that kind of usecases)

Re: How NASA built Artemis II’s fault-tolerant computer

#230

Post by mrheosuper »

gambiting wrote: post #134
BitsAndObjects wrote: post #127

Earlier quoted context omitted.

They’re not mission-critical equipment. If they fail, nobody dies. They’re not radiation hardened, so given enough time, they’d be expected to fail. Rebooting them might clear the issue or it might not (soft vs hard faults). Also impossible to predict when a failure would happen, but NASA, ESA and others have data somewhere that makes them believe the risk is high enough that mission critical systems need this level…

>>They’re not mission-critical equipment. If they fail, nobody dies. Yes, for sure, but that's not my question - it's not a "why is this allowed" but "why isn't this causing more visible problems with the iphones themselves". Like, do they need constant rebooting? Does this cause any noticable problems with their operation? Realistically, when would you expect a consumer grade phone to fail in these conditions?

A lot of "space-rated" components come from consumer space, with certification that it can work in space.

IIRC the Helicopter on Mars using the same snapdragon CPU in your phone.

Also, bit flip can happen without you knowing. A flip in free ram, or in a temp file that is not needed anymore won't manifest into any error, but then, your system is not really deterministic anymore since now you rely on chance.

Post reply on HN