We deleted the production database by accident • Hacker News Board

Re: We deleted the production database by accident

#261

Post by watwut » Sun, Oct 18, 2020, 9:15 AM UTC

cle wrote: post #90

danellis wrote: post #77

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

> Most stupid things are done whilst sober.

That is because most companies these days have processes around drinking in workplace, coming in drunk and working drunk.

Most mistakes are done sober only in environments where drinking couple of vine cups and then doing production change is considered unacceptable. In environment where drunk people work, mistakes are made when drunk.

Re: We deleted the production database by accident

#262

Post by watwut » Sun, Oct 18, 2020, 9:18 AM UTC

t0mas88 wrote: post #243

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

Yeah, but "pilot was drinking alcohol" would be considerate issue, would lead to fired pilot and would lead to more alcohol testing.

I understand what you are taking about, but aviation has also strong expectations on pilots.

Re: We deleted the production database by accident

#263

Post by kungtotte » Sun, Oct 18, 2020, 9:19 AM UTC

Agentlien wrote: post #228

kungtotte wrote: post #206

Earlier quoted context omitted.

Maybe we should, as a society, invest in taxis equipped with medical facilities and trained personnel so that they can provide first response medical treatment while on the way to the ER. I'm sure that would save a lot of lives. An ambulatory medical service, if you will.

Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening. I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours be…

Sure, that's true.

In that scenario it would also be appropriate to wait for a driver to sober up before driving you to the hospital if neither ambulance nor taxi were available (or delayed). One glass of wine would be out of most people's systems after two hours.

Thus poking hole in the "drunk drive someone to the hospital" argument, which is what this was all about in the first place.

Re: We deleted the production database by accident

#264

Post by Aeolun » Sun, Oct 18, 2020, 9:19 AM UTC

Agentlien wrote: post #228

kungtotte wrote: post #206

Earlier quoted context omitted.

Maybe we should, as a society, invest in taxis equipped with medical facilities and trained personnel so that they can provide first response medical treatment while on the way to the ER. I'm sure that would save a lot of lives. An ambulatory medical service, if you will.

Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening. I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours be…

That’s a problem with the ambulance service. Not with people being able to drive while drunk.

Re: We deleted the production database by accident

#265

Post by neillyons » Sun, Oct 18, 2020, 9:20 AM UTC

t0mas88 wrote: post #243

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

Re: We deleted the production database by accident

#266

Post by Aeolun » Sun, Oct 18, 2020, 9:21 AM UTC

danjac wrote: post #260

cranekam wrote: post #177

Earlier quoted context omitted.

The whole piece has a slightly annoying flippant tone to it. We were drunk! Computers just.. do this stuff sometimes! Better to sound contrite and boring in such a situation IMO. Also I agree with other comments: doing some work after a glass or two should be fine because you should have other defences in place. “Not being drunk” shouldn’t be the only protection you have against disaster.

The Exxon Valdez comes to mind - the company blamed the drunk captain, but this was just part of huge systemic failures and negligence.

If your captain feels the need to be drunk, you’ve probably made a few mistakes before it got to that point.

Re: We deleted the production database by accident

#267

Post by jurre » Sun, Oct 18, 2020, 9:22 AM UTC

qz2 wrote: post #259

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

You should have zero fear instilled when pressing any button. The system or process has failed if a single person pressing a button can bring something down unintended. Fix the system/process, don’t “instill fear” onto the person, it’s toxic, plus now you have to make sure any new person on boarded has “fear instilled”, and that’s just hugely unproductive

Re: We deleted the production database by accident

#268

Post by qz2 » Sun, Oct 18, 2020, 9:24 AM UTC

jurre wrote: post #267

qz2 wrote: post #259

Earlier quoted context omitted.

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

You should have zero fear instilled when pressing any button. The system or process has failed if a single person pressing a button can bring something down unintended. Fix the system/process, don’t “instill fear” onto the person, it’s toxic, plus now you have to make sure any new person on boarded has “fear instilled”, and that’s just hugely unproductive

That’s precisely my point. A lot of people have no fear because they’re complacent or ignorant rather than the button is well engineered.

But to get there you need to fear the bad outcomes.

Re: We deleted the production database by accident

#269

Post by janoc » Sun, Oct 18, 2020, 9:30 AM UTC

t0mas88 wrote: post #243

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It is not only that but also realizing that there is never a single cause to an accident or incident.

Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc.

So even when the accident is ultimately caused by a pilot's actions, there is always a chain of events where if any of the segments were broken the accident wouldn't have happened.

While we can't prevent a bonkers pilot from crashing a plane, we could perhaps prevent a bonkers crew member from flying the plane in the first place.

Aka the Swiss cheese model. You don't want to let the holes to align.

This approach is widely used in accident investigations and not only in aviation. Most industrial accidents are investigated like this, trying to understand the entire chain of events in order that processes could be improved and the problem prevented in the future.

Oh and there is one more key part in aviation that isn't elsewhere. The goal of an accident or incident investigation IS NOT TO APPORTION BLAME. It is to learn from it. That's why pilots in airlines with a healthy safety culture are encouraged to report problems, unsafe practices, etc. and this is used to fix the process instead of firing people. Once you start to play the blame game, people won't report problems - and you are flying blind into a disaster sooner or later.

Re: We deleted the production database by accident

#270

Post by janoc » Sun, Oct 18, 2020, 9:32 AM UTC

neillyons wrote: post #265

t0mas88 wrote: post #243

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

Not sure about books but look up the Swiss cheese model. It is widely used approach and not only in aviation. Most industrial accidents and incidents are investigated with this in mind.