Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

301–310 of 456 posts

Re: We deleted the production database by accident

#301

Post by corobo »

qz2 wrote: post #259
skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

Most will hide it away because being truthful will hurt current business or future career prospects because people like yourself exist who want everyone shitting themselves at the prospect of being honest.

In a blame free environment you find the underlying issue and fix it. In a blame full environment you cover up the mistake to avoid being fired and some other person does it again later down the line

Re: We deleted the production database by accident

#302

Post by qz2 »

corobo wrote: post #301
qz2 wrote: post #259

Earlier quoted context omitted.

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

Most will hide it away because being truthful will hurt current business or future career prospects because people like yourself exist who want everyone shitting themselves at the prospect of being honest. In a blame free environment you find the underlying issue and fix it. In a blame full environment you cover up the mistake to avoid being fired and some other person does it again later down the line

No.

There’s a third option where people accept responsibility and are rewarded for that rather than hide from it one way or another.

I have a modicum of respect for people who do that. I don’t for people who shovel it under a rug or point a finger which are the points you are referring to. I’ve been in both environments and neither end up with a positive outcome.

If I fuck up I’m the first person to put my hand up. Call it responsibility culture.

Re: We deleted the production database by accident

#303

Post by aszen »

heavenlyblue wrote: post #237
aszen wrote: post #184

I had a narrow escape once doing something fancy with migrations. We had several MySQL string columns as long text type in our database but they should have been varchar(255) or so. So I was assigned to convert these columns to their appropriate size. Being the good developer I was, I decided to download a snapshot of the prod database locally and checked the maximum string length we had for each column via a script.…

And you didn’t even bother to do a query of the actual maximum length value of the columns you were mutating? Or at least query and see the text in there? Basically you just blindly ran the migration on the data and checked if it didn’t fail? The lesson here is not about cleverness unfortunately.

I did see some values and found them reasonable, problem of the whole thing was there were atleast 200 or so tables with dozens of columns each and only two or so tables were excluded from being dumped locally.

So yes I could have noticed their length 0 if I had looked carefully amidst hundreds of rows but since my faulty logic of prod db = local db didn't even consider this possible I didn't bother.

If it had been just 10 to 20 migrations queries that would have been a lot easier to validate but then I wouldn't even have attempted to write a script

Re: We deleted the production database by accident

#304

Post by dschuetz »

Someone might have copy&pasted it elsewhere and that propagated away. This is why writing code can also be dangerous in open dev. Whoever programmed anything also should be sensible enough to judge their own code whether it could be dangerous in the wild. Once out there (or worse: on stackoverflow) it could wreak havoc.

Re: We deleted the production database by accident

#305

Post by MaxBarraclough »

qz2 wrote: post #259
skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

You're speaking to the mistake. The comment you're replying to is speaking to the write-up analysing the mistake.

Blog posts analysing real-world mistakes should not be met with beratement.

Re: We deleted the production database by accident

#306

Post by ordu »

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

> I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault".

It seems that people annoyed mostly by "complexity gremlins". They are so annoyed, that they miss previous sentence "we’re too tired to figure it out right now." Guys fucked up their system, they restored it the best they could, they tried to figure out what happened, but failed. So they decided to do PR right now, to explain what they know, and to continue the investigation later.

But people see just "complexity gremlins". The lesson learned is do not try any humor in a postmortem. Be as serious, grave, and dull as you can.

Re: We deleted the production database by accident

#307

Post by dschuetz »

unnouinceput wrote: post #199

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

As someone else said it below: "hardcoded to localhost" doesn't mean it's hardcoded. It means it goes to whatever localhost resolves to. Really hardcoded should ALWAYS mean: 127.0.0.1

Re: We deleted the production database by accident

#308

Post by auggierose »

thunderrabbit wrote: post #213
ikiris wrote: post #193

Earlier quoted context omitted.

excuse me, but no. this is harmful bullshit. Yes, backups are vitally important, but no it is not possible to accidentally rm -rf with proper design. It's possible to have the most dangerous credentials possible and still make it difficult to do catastrophic global changes. Hell it's my job to make sure this is the case.

> not possible to accidentally rm -rf with proper design. Can you say more about this? I understand rm -rf, but not sure how I could design that to be impossible for the most dangerous credentials.

He said "difficult", not impossible.

Re: We deleted the production database by accident

#309

Post by aszen »

heavenlyblue wrote: post #289
detaro wrote: post #238

Earlier quoted context omitted.

The comment clearly states that they did.

If they did, they would have noticed that the columns were empty (because they were wiped clean for PI data). The parent is either misrepresenting the situation or they didn’t do what they say they did. Also in any production setup, before the migration in the same transaction you would have something along the lines of “check if the column size is larger than and then abort”, because you never know when that can be…

I agree we could have done it a lot differently and safely, I especially like the last point you mentioned that is what would have been the correct way to do it.

But this happened as described, a local only script that generated a list of columns to modify then a migration to execute the alter queries for all of them.

Re: We deleted the production database by accident

#310

Post by madbkarim »

heavenlyblue wrote: post #299
auroranil wrote: post #222

Earlier quoted context omitted.

Tom Scott made a mistake with a similar outcome as this article, but with an SQL query that is much more subtle than DROP. https://www.youtube.com/watch?v=X6NJkWbM1xk By all means, find ways to fool-proof the architecture. But be prepared for scenarios where some destructive action happens to a production database.

He would not have done that if he were simply using a database transaction for this operation.

That’s exactly the point he’s trying to get across with that video.
Post reply on HN