Live data from Hacker News

We deleted the production database by accident

keepthescore.co

241–250 of 456 posts

Re: We deleted the production database by accident

#241
post #78

I love this post. This sort of thing happens to everyone, most people just are not willing to be so open about it. I was once sshed to the production server, and was cleaning up some old files that got created by an errant script, one which file was '~'. So, to clean it up, I type `rm -rf ~`.

Somewhat similar story from many years ago. Was in ~/somedirectory, wanted to clear the contents, ran `rm -rf *`. Turns out somewhere in between I had done a `cd ..`, but I thought I was still in the child directory. Fastest Ctrl+C ever once I saw some permission errors, but most of the home directory was wiped in that second or two. Didn't have a backup of it unfortunately, though thankfully there wasn't anything to…

It would be beneficial for `rm` to have a way for making `trash` mode the default mode of operation. Unix shell is prone to errors like this beyond all reason. And nobody would die typing:

  fs.delete(shell.expand('*'), recursive:yes)
  fs.undo()
or something like that with completion helpers. Our instruments usually lack any safety in general, and making them safe is hard, especially when you're just a worker and not a safety expert. All the world today benefits from mistake-friendly software, except for developers who constantly walk on ui minefields.

Re: We deleted the production database by accident

#243

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a huge improvement in safety across the whole industry.

In software there is still a certain arrogance of quickly calling the user (or other software professional) stupid, thinking it can't happen to you. But in reality given enough time, everyone makes at least one stupid mistake, it's how humans work.

Re: We deleted the production database by accident

#244

Earlier quoted context omitted.

There is sense in not building more services than you need. But many folks end up finding it hard to break away from their monolith and then it becomes an albatross. Not sure how to account for that.

If a team doesn’t have the engineering know how to split a monolith into distinct services in a straightforward fashion, then I’m not sure that team will have the chops to start with microservices.

What you end up with is a set of tiny monoliths.

Re: We deleted the production database by accident

#245

I wouldn’t want to be on the wrong side of a lawsuit, defending drunk employees working on production data. What outrageous recklessness. And how imprudent to admit this to the public. Some things are best kept to yourself. No one needs to know that.

This is someone's side project. I'm doubtful there will be any lawsuits involved

Re: We deleted the production database by accident

#246

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

Re: We deleted the production database by accident

#247

The wonderful thing about computers is that they do exactly what they are told to do. The worst thing about computers? They do exactly what they are told to do.

they do exactly what they are told to do

Well, tell them to UNDROP and see what happens.

Re: We deleted the production database by accident

#249

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

2 cents his hosts file points localhost to the prod db IP

One solution to this is to make sure only the production servers can connect to the production database.

Re: We deleted the production database by accident

#250

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

It isn't about the importance of this one database, it's about the cultural issue in most of the sector that the parent comment was pointing out: we far too often blame the user/operator calling them stupid, while every human makes mistake, it's inevitable.
Post reply on HN