Live data from Hacker News

We deleted the production database by accident

keepthescore.co

321–330 of 456 posts

Re: We deleted the production database by accident

#321

Earlier quoted context omitted.

Yes. I could finally start a failed interlock story blog.

Or a blog on being unable to drive your kid to an emergency room because you just finished a glass of wine over dinner. A problem with devices of that type is that they only test for a potential source of inability to drive safely. What we want is to test for an inability to drive safely. And while one is easy and might give some quick wins, the drawbacks scare me too much.

[deleted]

Re: We deleted the production database by accident

#322

Earlier quoted context omitted.

Similar effect with psilocybin or LSD - you think you had a really profound and insightful experience, but once you think back on it you realize that (most of the time) you just got the impression that it was profound and insightful.

Its there any difference between having a profound experience and having "just the impression" of it? Also, nothing is comparable between alcohol and psychodelics

[deleted]

Re: We deleted the production database by accident

#323
post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

Replace a breathalyzer with something that's less intruisive (like a camera with AI that would observe the person, AI with thermal imaging or air quality sensors, or another possibly-fictional-yet-believable piece of technology) and suddenly, in my eyes, the technology in this thought experiment becomes a no brainer.

If there was more of a cultural pressure against drunk driving and actual mechanisms to prevent it that aren't too difficult to maintain and utilize, things like the Daikou services ( https://www.quora.com/How-effective-is-the-Japanese-daikou-s... ) would pop up and take care of the other logistical problems of getting your car home. And the world would be all the better for it, because of less drunk driving and accidents.

Re: We deleted the production database by accident

#324
post #269

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

> Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc.

There was indeed a suicidal pilot that flew into a mountain, I'm not sure if you were deliberately referencing that specific time. In that case he was alone in the cabin – this would have happened briefly but he was able to lock the cabin door before anyone re-entered, and the lock cannot be opened by anyone from the other side in order to avoid September 11th-type situations. It only locks for a brief period but it can be reapplied from the pilot side before it expires an indefinite number of times.

I'm not saying that we can put that one down purely to human action, just that (to be pedantic) he wasn't being supervised by anyone, and there were already any number of alarms going off (and the frantic copilots on the other side of the door were well aware of them).

Re: We deleted the production database by accident

#325
post #302
post #301

Earlier quoted context omitted.

Most will hide it away because being truthful will hurt current business or future career prospects because people like yourself exist who want everyone shitting themselves at the prospect of being honest. In a blame free environment you find the underlying issue and fix it. In a blame full environment you cover up the mistake to avoid being fired and some other person does it again later down the line

No. There’s a third option where people accept responsibility and are rewarded for that rather than hide from it one way or another. I have a modicum of respect for people who do that. I don’t for people who shovel it under a rug or point a finger which are the points you are referring to. I’ve been in both environments and neither end up with a positive outcome. If I fuck up I’m the first person to put my hand up. C…

I think you missing point, I love the idea about "bringing aviation methodology" to lower error/f-up-rates" for the software industry.

No one is not saying don't take responsibility, they are saying - as I understood it:

Have a "systematic-approach" to the problem, the current system for preventing "drunk pilots or the wiping of production db's are not sufficient" - improve the system ! ! All the "taking responsibilities and "falling on one's own sword" won't improve the process for the future.

If we take the example of the Space-Industry where having 3x Backups Systems are common (like life support)

It seems some people's view in the comments stream is:

"No bugger that, the life-support systems engineers and co should just 'take responsibility' and produce flawless products. No need for this 3 x backups systems"

The "system" approach is that there is x-rates of failures by having 2 backups we have now reduce the possibility of error by y amount.

Or in the case of production-dbs:

If I were the CEO and the following happens:

CTO-1: "Worker A, has deleted the production DB, I've scolded him, he is sorry and got dock a months pay and is feeling quite bad and has taken full responsibility for his stupid action this probably won't happen again !"

VS

CTO-2: "Worker A, has deleted the production DB, We Identified that our process/system for allowing dev-machines to access production db's was a terrible idea and oversight, we now have measures abc in place to prevent that in the future"

I'd go with CTO-2 EVERY day of the week !

Re: We deleted the production database by accident

#326
post #312
post #311

Earlier quoted context omitted.

See that leading with a "No." there That's what we're talking about. I hope you don't have direct reports. Next time be honest "Just shut the conversation down, everyone's a dumbass, I'm right, you're dumb" it'll be quicker than all this back and forth actually trying to get to a stable middle ground :)

That's a tad ironic is it not? All I am calling for is people to take responsibility.

What is the future value in that ? From a system and reliability point of view ? Genuine question - not trying to be a dk

Re: We deleted the production database by accident

#327

Earlier quoted context omitted.

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

When I was getting my pilots license I used to read accident reports from Canada's Transportation Safety Board [1]. I'm sure the NTSB (America's version) has similar calibre reports [2]. There is also Cockpit Resource Management [3] which addresses the human factor in great detail (how people work with each other, and how prepared are people). In general what you learn from reading these things is that its rarely one…

The old "they write the right stuff" essay on the On-Board Shuttle Group also talked about this mindset of errors getting through the process as being first and foremost a problem with the process to be examined in detail and fixed.

Re: We deleted the production database by accident

#328

This post is embarrassing. "yeah we were drinking and accidentally nuked the prod DB. Not sure why. Shit happens!" Who would read this and think they should trust this company? Any number of protections could have been taken to prevent this and production access in any state other than fully alert and attentive shouldn't happen unless it is absolutely necessary for emergency reasons

This reply is embarrassing. It's a person working on their side project. Have a glass of wine mate.

Re: We deleted the production database by accident

#329

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

Anything by Sidney Dekker. https://sidneydekker.com/books/ I would start by The Field Guide to Unterstanding 'Human Error'. It's very approachable and gives you a solid understanding of the field.

Re: We deleted the production database by accident

#330
post #318

Earlier quoted context omitted.

>It really depends on your database size. This isn't just about size though. You're storing all customer data on all developer machines. You're just one stolen laptop away from your very own "we take the security of our customers' data very seriously" moment.

Nobody said the devs have access to those repos

True. It depends on how exactly those repositories are structured and managed. Hopefully it's not quite as bad as I imagined.

I still think that database size is not the only consideration.

Post reply on HN