NASA can't figure out what's causing computer issues on the Hubble telescope • Hacker News Board

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#151

Post by kevmo » Thu, Jun 24, 2021, 4:35 PM UTC

kabdib wrote: post #124

qzw wrote: post #7

pushes up glasses I would watch the heck out of a Twitch stream of their debugging/brainstorming sessions. I always loved the movie Apollo 13 , especially the technical troubleshooting parts.

Henry S F Cooper Jr.'s book The Evening Star describes some of the remote debugging and other problem solving that was necessary when the Magellan probe experienced computer problems while orbiting Venus. It's been a few decades since I read it, but it was pretty detailed and rather exciting.

This sort of comment is why I still read HN.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#152

Post by ComputerGuru » Thu, Jun 24, 2021, 4:37 PM UTC

londons_explore wrote: post #129

35fbe7d3d5b9 wrote: post #107

Earlier quoted context omitted.

One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone else to stop what they are doing. The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.

In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that…

> If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.

Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#153

Post by LadyCailin » Thu, Jun 24, 2021, 4:38 PM UTC

woodruffw wrote: post #149

IAmMaulik wrote: post #141

>"They're very primitive computers compared to what's in your cell phone," I do not understand why they can't just swap out the computer for a better and more modern one. Am I missing something here?

> Am I missing something here? When dealing with high-latency, high-radiation environments, more modern isn't necessarily better: denser ICs mean greater susceptibility to radiation (and consequently more expensive hardening). They also can't exactly fly up there and swap out random bits on short notice -- I'm not sure if the US even has a the current capability to perform physical maintenance on the Hubble.

They don't.

> Hertz said that because Hubble was designed to be serviced by the space shuttle and the space shuttle fleet has since been retired, there are no future plans to service the outer space observatory.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#154

Post by ComputerGuru » Thu, Jun 24, 2021, 4:40 PM UTC

mikeytown2 wrote: post #9

The cause of the failure is most likely tin whiskers [1] or radiation. [1] https://nepp.nasa.gov/WHISKER/background/index.htm .

I thought NASA no longer used lead-free solder to avoid this.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#155

Post by hungryforcodes » Thu, Jun 24, 2021, 4:41 PM UTC

afterburner wrote: post #100

qzw wrote: post #7

pushes up glasses I would watch the heck out of a Twitch stream of their debugging/brainstorming sessions. I always loved the movie Apollo 13 , especially the technical troubleshooting parts.

I think what you're really saying is you'd watch the movie version of this. And considering there are no life or death stakes, it still wouldn't be as exciting as Apollo 13.

That might not be true though.

This guy took about 30 hours of video of him porting an 80s version of unix to the ESP8266. Warts and all -- live!

I've started to watch it and it's fascinating!

https://www.youtube.com/watch?v=cDHcGY7EzUM&t=62s

You could have a whole channel with different teams debugging satellite technology and if you're bored, it would probably be quite interesting. The bigger problem is most likely concerns about IP and secret protocols and so on.

"And now Bob will log into TeleSat123 via SSH."

"Oups, uh..gosh we'll just go to a commercial break!"

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#156

Post by dehrmann » Thu, Jun 24, 2021, 4:43 PM UTC

qzw wrote: post #7

pushes up glasses I would watch the heck out of a Twitch stream of their debugging/brainstorming sessions. I always loved the movie Apollo 13 , especially the technical troubleshooting parts.

I have no idea if it would be more or less exciting than a tech company warroom.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#157

Post by bumby » Thu, Jun 24, 2021, 4:44 PM UTC

trothamel wrote: post #120

qzw wrote: post #7

pushes up glasses I would watch the heck out of a Twitch stream of their debugging/brainstorming sessions. I always loved the movie Apollo 13 , especially the technical troubleshooting parts.

If you want to see debugging a computer in space, check out Apollo 13's sequel, Apollo 14. The moon landing is being held up by shorted-out switch that's causing the LM to abort the landing, and it's up to the programmers back home to figure out how to work around it in time to allow the landing. Apollo 13 was the story of a 'successful failure', while Apollo 14 shows how hard work and creative thinking can turn fail…

Was this the scenario where there was a false positive warning light but they had no way to test if it was truly a false alarm? I remember attending a talk by an Apollo engineer who convinced the control room that the switch design had a propensity to a short and it really came down to a probability-based judgement call

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#158

Post by peter303 » Thu, Jun 24, 2021, 4:46 PM UTC

thrower123 wrote: post #69

Maybe the James Webb will go up some day...

Scheduled for Halliween this year, but could slip.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#159

Post by dylan604 » Thu, Jun 24, 2021, 4:47 PM UTC

Koshkin wrote: post #37

dylan604 wrote: post #22

Earlier quoted context omitted.

What if you backup is failing in the exact same way?

That would probably mean that the whole thing just isn't there anymore.

Or much more likely the same component was used as a back up, and is failing in a similar fashion. It's obvious the thing is still there.

Re: NASA can't figure out what's causing computer issues on the Hubble telescope

#160

Post by ignoranceprior » Thu, Jun 24, 2021, 4:50 PM UTC

programmer_an wrote: post #29

Programmer/analyst here and ready to help you NASA, just ask me and I'll clear a few hours of my agenda for you. I know people say this a lot, but in this case I really think a (at least partial) rewrite in Rust of the Hubble software would be very beneficial. We could gather some of the most distinguished coders here in hacker-news and create a task force to show them the benefits of rust's memory safety.

Not sure if serious or joking. Poe's law.