71–80 of 361 posts
The roots cause, and someone correct me if this is not accurate, was that the x-ray tested bolts to hold it down were so expensive, that they had been "borrowed" to use on another project, and not returned, so that when the time came to flip the satellite into a horizontal position, it fell to the floor. Repairs cost $135M.
I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…
PS: Also, more rules and better processes are not necessarily a good thing. Sometimes there are just too much red tape and bureaucracy that makes already super-slow NASA even slower. In those first-of-its-kind missions sometimes you need to risk and depend on people, not processes.
Earlier quoted context omitted.
Another relevant bit of info from hospital accidents: hand-offs between shifts are known to increase the risk of a mistake in care and are part of the reason nurses and doctors work such long hours.
I avoid, if possible of course, going to the hospital right before a shift change for this very reason.
Earlier quoted context omitted.
I have bad news for you about health care professionals.
Health care professionals is a weird one, because while long shifts are dangerous, patient handover is also dangerous and there may be an argument that longer shifts means fewer handovers which could result in better patient outcomes.
Earlier quoted context omitted.
You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…
You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not lett…
Earlier quoted context omitted.
> work is too important to work 12 hour shifts on it Yes. Because it is know that exhausted people make mistakes. The work is too important to let exhausted people screw it up so you should make sure everyone working on it is well rested. > your solution is throwing “cheap” grad students at it Yes? It is testing an electric motor. They can do it. The solution is that you employ enough people so nobody needs to work 1…
That “solution” is nothing more than typical HN backseat driving. In the real world there are budget, personnel and hiring constraints. You don’t get to hire all the people you want. You make do with what you have, and try to push the mission forward, even in suboptimal conditions.
Reminds me of a quote attributed to Thomas J Watson: Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?
THE LITTLE VAX THAT COULD https://userpages.umbc.edu/~rostamia/misc/vax.html
I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…
At my first real job as a web dev after school, I crashed the production website on my very first day. Tens of thousands of visitors were affected, and all our sales leads stopped.
Thankfully, we were able to bring it back up within a few minutes, but it was still a harrowing ordeal. The entire team (and the CEO in the next room) was watching. It ended up fine and we laughed about it after some minor hazing :)
But by the time I left that job a couple years later, we had turned that fragile, unstable website into something with automatic testing, multiple levels of backups and failover systems across multiple data centers, along with detailed training and on-boarding for new devs. (This was in the early days of AWS, and production websites weren't just a one click deploy yet.)
That one experience led to me learning proper version control, dev environments, redis, sharding and clustering, VMs, Postgres and MySQL replication, wiki, monit, DNS, load balancers, reverse proxies, etc. All because I was so scared of ever crashing the website again.
That small company took a chance on me, a high school dropout with some WordPress experience, and paid me $15/hour to run their production website, lol. But they didn't fire me after I screwed up, and gave me the freedom and trust to learn on the job and improve their systems. I'm forever grateful to them!