Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

331–340 of 361 posts

Re: My $500M Mars rover mistake

#331

Post by gosub100 »

lostlogin wrote: post #318
zimpenfish wrote: post #306

Earlier quoted context omitted.

As I said on Slack the other day in response to a similar story, "If, on your first day, you can destroy the prod database, it's not your fault." (One of my standard end-of-interview questions is "how easy is it for me to trash the production database?" Having done this previously[1] and had a few near misses, it's not something I want to do again.) [1] In my defence, I was young and didn't know that /tmp on Solaris…

> /tmp on Solaris was special. I’ve had a search but can’t work out why it’s special.

it gets wiped on reboot. I remember around 2007 on Gentoo Linux, this behavior changed. I was using /tmp as pretty much a "my documents" type folder, I updated, and one day all my stuff was gone! I was flabbergasted. But yeah, it was reckless to store things on a folder that pretty has "temp" in the name!

Re: My $500M Mars rover mistake

#332

Post by johnwalkr »

i-use-nixos-btw wrote: post #294

I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them cus…

I work in this industry and let me explain how this happens. Despite being such a costly project, you can’t really hard-require unique connectors everywhere because of all of the competing requirements. Actually, connectors in particular tend to have a lot of conservative requirements such as being previously qualified, certain deratings, pin spacing, grounded back shells, etc. At the end of the day there’s only a handful of connector series used and stocked and it’s not feasible (at any cost really) to have no matching connectors whatsoever. Of course, you would normally try and make connectors either standardized with the same signals, or unique with no overlap in between.

I don’t know the details in this case but it could be like this: socket-type connectors are required on external connectors on the spacecraft (to prevent shorts when handling), with a harness in between which will never be removed. The harness would be symmetrical with pin-type connectors.

At some point it is decided a breakout box is required for testing and now you have created an opportunity to plug the breakout box in backwards.

Or the breakout box has a 100 pin connector on one side and needs to connect to 25 pieces of test equipment on the other side. You probably don’t have 25 different connectors to chose from, nor can you possibly demand custom requirements for every piece of test equipment.

Spacecraft are moving more towards local microcontrollers with local diagnostics so this kind of test equipment for every possible analogue signal is decreasing. In the case of motors, they would more likely be brushless now and you would rely on telemetry from motor drivers during both testing and flight instead of having this type of breakout box.

Connectors in aerospace are also following other industries and becoming more configurable at order time, including adding keys so you can have 10x “the same” connector but keyed so they only plug in one place. But it’s still not practical to demand all test equipment is configured like this.

Re: My $500M Mars rover mistake

#333

Post by [deleted] »

initplus wrote: post #96

Interesting to see that the worry could have been avoided if they had lined up their timelines better in the first place. If they'd compared the timestamp on the test readout to the last timestamp from the telemetry system, they'd have seen that the telemetry failed BEFORE the test was executed. Partially caused by using imprecise language "we seem to have lost all spacecraft telemetry just a bit ago" rather than an…

[deleted]

Re: My $500M Mars rover mistake

#334

Post by wolverine876 »

cpgxiii wrote: post #329
wolverine876 wrote: post #187

Earlier quoted context omitted.

Without disagreeing with your point, would availability be an issue in this case? They need one or two, have an enormous budget, and if the technology exists can make their own.

Availability is often strongly correlated with technical maturity. Small brushless motors with FOC didn't become widely available and mature until really the late 2010s. Arguably the foundation of nearly all of DJI's product lines is due to their early mastery of small brushless motor control (drones, gimbals, lens controls, robots, etc), and that's a company founded in 2006, well after the events of the article. You…

> Availability is often strongly correlated with technical maturity.

I see what you mean. Yes, agreed.

Re: My $500M Mars rover mistake

#335

Post by dgunay »

typpo wrote: post #87
dgunay wrote: post #32

One thing about long aerospace missions like this with huge lead times that always gets me - you can spend years of your life working on a mission, only for it all to fail with potentially years until you can try again. This is a refreshingly humanizing article, but is also one written from the perspective of a survivor. Imagine if the rover were actually lost. I asked the question "what would you do if the mission f…

> you may literally see your life's work go up in flames. Incidentally, this happened to Lewicki a few years later when Planetary Resources' first satellite blew up on an Antares rocket: https://www.geekwire.com/2014/rocket-carrying-planetary-reso...

Did they have a narrow launch window they couldn't afford to miss? I'm not talking about missions where you eat a big monetary loss on the launchpad and try again, I mean missions which rely on planetary alignments that may not happen again for years, or even the rest of your life, such as Voyager. Or even just missions where you launch successfully, but then after months (or years) of flight time the spacecraft is lost.

Re: My $500M Mars rover mistake

#336

Post by technimad »

Feathercrown wrote: post #169
markmark wrote: post #160

Earlier quoted context omitted.

At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each…

Isn't that almost exactly what happened at github too?

This happened to Gitlab.

Re: My $500M Mars rover mistake

#337

Post by deepsun »

skrebbel wrote: post #193
deepsun wrote: post #62

Earlier quoted context omitted.

> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.

Firing people for incidental mistakes instead of overall bad performance is pretty shitty management.

For one incidental mistake of course not. For repeated inattention (like plugging mars rover's cables wrongly several times) at an attention-demanding job -- yes.

Re: My $500M Mars rover mistake

#338

Post by spixy »

norman784 wrote: post #211
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Our test suite expects that the database name has a `_test` suffix, so you can't run the tests even locally without the suffix.

Our test suite uses DB user that exists in docker DB but not in prod, so droping prod database cannot happen.

Re: My $500M Mars rover mistake

#339

Post by vichle »

cdogl wrote: post #92
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

This is why I always delete by ID when cleaning up after tests.
Post reply on HN