Earlier quoted context omitted.
They fired you for that array_reverse mistake?
Yep, I was put on PIP with impossible success criteria (no issues raised in PRs by senior engineers and no issues in code deployed to production - even if it was reviewed by senior engineers & QA) and fired (for failing that criteria) in 2 weeks. I worked there for ~8 months in total.
My $500M Mars rover mistake
181–190 of 361 posts
Re: My $500M Mars rover mistake
#182Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…
Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update…
Re: My $500M Mars rover mistake
#183"Today is the safest day to let you drive my truck, cause I know you'll be extra careful"
Re: My $500M Mars rover mistake
#184I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…
At a major brokerage firm I accidentally hit prod with a testing script that did several millions of dollars of fake FX test trades. The first thing mentioned in the post mortem call was “No one is going to blame the guy who did those trades. It was an honest mistake. What we are going by to do is discuss why a developer can hit the production trading API without any authentication at all”.
Re: My $500M Mars rover mistake
#185I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…
> I'm instantly transported back to that moment — the room, the lighting, the chair I was in, the table, the pit in my stomach, ...
I could'nt help but think "that sounds like a trauma reaction". Good on them to be able to use that energy to do better! But also not everyone reacts the same way to trauma nor is it easy to compare such reactions to trauma (for example as a hiring question). I feel there are too many social variables at play
Re: My $500M Mars rover mistake
#186I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…
> I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. Does it inevitably come down to that for someone? I mean even if its a detail that a procedure couldn’t have caught, someone is responsible for forming good procedures. I suppose there could be several factors. But it seems like ultimately someone is going to be pretty directly resp…
Or many people, or no one directly. Space missions come with calculated risk. So someone calculates the risk that this critical part brakes is 0.5% and then someone higher up says, that is acceptable and all move on - and then this part indeed brakes and people die.
Who is to blame, when the calculation was indeed correct, but 0.5% chances still can happen (and itnwould be a lot)? And economic pressures are real, like the limits of physic?
See Murpheys Law, "Anything that can go wrong will go wrong." (Eventually, if done again and again)
https://en.m.wikipedia.org/wiki/Murphy's_law
Astronauts know, there is a risk with every mission, so do the engeneers, so does management. Still, I cannot imagine why anyone thought it was an acceptable risk, to use a 100% oxygen atmosphere with Apollo 1, where 3 Astronauts died in a fire. But that incident indeed changed a lot regarding safety procedures and thinking about safety. Still, some risk remain and you have to live with that.
I am quite happy though, that in my line of work, the worst that can happen is a browser crash.
Re: My $500M Mars rover mistake
#187Earlier quoted context omitted.
Any idea why they would use brushed motors? When every gram counts I would think ditching the mechanical commutator would be a no-brainer, but maybe adding another leg to the H-bridge is a bigger penalty?
In 2003, small brushless DC motors were way less mature and available as they are now, particularly for low-speed/high-torque applications*. Brushless controllers are much more complicated than brushed controllers, particularly on the control software front, so sticking with simpler and more reliable brushed controllers for a space application makes sense (remember, it probably needed to be radiation hardened - doing…
Re: My $500M Mars rover mistake
#188I used to work in a rural manufacturing plant. I once totaled my car on the way to town. Our shop foreman was also a volunteer firefighter and was the first one on the scene. I survived with a bloody nose and shattered confidence. The next day at work we needed some parts from town. I didn't have a car anymore, but that same shop foreman lent me his suped up pickup truck. I was totally confused. He wisely said: "Toda…
Re: My $500M Mars rover mistake
#189Bravo to the author. I think if I had made the same mistake, and if the disaster is permanent, I would probably be scarred for life and never recover from it.
Re: My $500M Mars rover mistake
#190Earlier quoted context omitted.
They fired you for that array_reverse mistake?
Yep, I was put on PIP with impossible success criteria (no issues raised in PRs by senior engineers and no issues in code deployed to production - even if it was reviewed by senior engineers & QA) and fired (for failing that criteria) in 2 weeks. I worked there for ~8 months in total.
Wat? Like serious issues, or minor things that can be improved? Because it's very rare in my place of work that there are no comments on a 'PR'. Something can always be improved.