Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

291–300 of 456 posts

Re: We deleted the production database by accident

#291

Post by Aeolun »

tcbasche wrote: post #196
emodendroket wrote: post #119

This post is embarrassing. "yeah we were drinking and accidentally nuked the prod DB. Not sure why. Shit happens!" Who would read this and think they should trust this company? Any number of protections could have been taken to prevent this and production access in any state other than fully alert and attentive shouldn't happen unless it is absolutely necessary for emergency reasons

Yeah why should I treat anything this company does with any level of seriousness? Why should anyone? It's lucky it's just some online scoreboard because I'm sure as shit this stuff has happened before with more critical systems and it scares the hell out of me that engineers are fine blaming "gremlins" instead of taking responsibility for their own incompetence.

> taking responsibility for their own incompetence.

I think they’re doing that with this post? At least I find it hard to imagine myself writing down that I’d drunk a few glasses of wine and dropped the production database.

You cannot expect all engineers to be fully versed in the vagarities of database administration. Especially if they’re the only ones working on something.

Re: We deleted the production database by accident

#292

Post by oblio »

theamk wrote: post #133
dnautics wrote: post #107

Earlier quoted context omitted.

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

Do use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have co…

> complex data (punctuation, newlines, etc..)

Oh, the irony! Text with punctuation and newlines is complex data.

CSV is doomed but the world runs on it and pays its cost with engineer tears.

Re: We deleted the production database by accident

#293

Post by DelightOne »

michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

If you use terraform to deploy the managed production database, do you use the postgresql terraform provider to create roles or are you creating them manually?

Re: We deleted the production database by accident

#294

Post by oblio »

t0mas88 wrote: post #243
skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

The user? Start a discussion about using better programming language and you'll see people, even here, blaming the developer.

The common example is C: "C is a sharp tool, but with a sufficiently smart, careful and experienced developer it does what you want (you're holding it wrong").

Developers still do this to each other.

Re: We deleted the production database by accident

#297

Post by nicoburns »

TedDoesntTalk wrote: post #93
cle wrote: post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

> Coding after drinking is probably not a good idea I’ve done some of my most productive work this way. Not on production systems fortunately, and not in a long time.

A highly experienced developer in their 50s who I used to work with said that they used to regularly sit down and code with a pint. Until on one occasion they introduced some Undefined Behaviour into their aplication during one of these sessions and it took them 3 days to track down! Probably less of an issue with modern tooling. Still, it certainly makes me think twice before drinking and coding.

Re: We deleted the production database by accident

#298

Post by Agentlien »

Aeolun wrote: post #264
Agentlien wrote: post #228

Earlier quoted context omitted.

Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening. I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours be…

That’s a problem with the ambulance service. Not with people being able to drive while drunk.

Yes. I was answering a comment suggesting the use of an ambulance (instead of a regular taxi). Simply pointing out that, in practice, there are times when a taxi can get you there faster.

Re: We deleted the production database by accident

#299

Post by heavenlyblue »

auroranil wrote: post #222
michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

Tom Scott made a mistake with a similar outcome as this article, but with an SQL query that is much more subtle than DROP. https://www.youtube.com/watch?v=X6NJkWbM1xk By all means, find ways to fool-proof the architecture. But be prepared for scenarios where some destructive action happens to a production database.

He would not have done that if he were simply using a database transaction for this operation.

Re: We deleted the production database by accident

#300

Post by heavenlyblue »

thunderrabbit wrote: post #213
ikiris wrote: post #193

Earlier quoted context omitted.

excuse me, but no. this is harmful bullshit. Yes, backups are vitally important, but no it is not possible to accidentally rm -rf with proper design. It's possible to have the most dangerous credentials possible and still make it difficult to do catastrophic global changes. Hell it's my job to make sure this is the case.

> not possible to accidentally rm -rf with proper design. Can you say more about this? I understand rm -rf, but not sure how I could design that to be impossible for the most dangerous credentials.

Just don’t use the most dangerous credentials.

The most dangerous credentials are cosmic rays and we use the Earth’s atmosphere and ECC to fight that.

Post reply on HN