Live data from Hacker News

We deleted the production database by accident

keepthescore.co

101–110 of 456 posts

Re: We deleted the production database by accident

#101
post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

[deleted]

Re: We deleted the production database by accident

#102
post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

Don’t be absurd. Of course there are costs and tradeoffs to guardrails, and you have to balance the tradeoffs based on your requirements.

This person had to publicly apologize to their customers. One or two low-cost guardrails could have prevented it and would probably have been worth the cost.

Re: We deleted the production database by accident

#103
post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

> Coding after drinking is probably not a good idea I’ve done some of my most productive work this way. Not on production systems fortunately, and not in a long time.

Yes, it can be productive (depends on exactly what you are doing). But I imagine you revise your work while sober before you deploy it.

Re: We deleted the production database by accident

#104
post #29
post #16

This is my greatest fear when it comes to terraform: > terraform destroy (And either a confirmation or a flag) and everything is deleted. I know you can add some locks but still :/

You can save yourself from scary operations like deleting everything by a.) not rooting your entire infra in the same main.tf and b.) using Terraform's lifecycle meta-argument: https://www.terraform.io/docs/configuration/resources.html#l... I like to use the lifecycle feature for suuuper core things that will never be deleted (VPC, r53 zone, etc) and eventually when I start targeting multiple DCs w/ lots of infra I'l…

Also a human should review every plan and confirm before applying, right?

Re: We deleted the production database by accident

#105

Earlier quoted context omitted.

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

Re: We deleted the production database by accident

#106
I did the same thing once by accident and thankfully only lost 1 hour of data. The single lowest point of my career and thinking about the details of that day makes my stomach sink even now as I type this.

I ran a rake db schema dump command of some kind and instead of it returning the schema of the database, it decided to just completely nuke my entire database. Instantly. It's very easy to fuck up so cover your ass gents, and backup often and run your restores periodically to make you can actually do them in case of an emergency.

Re: We deleted the production database by accident

#107

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

Re: We deleted the production database by accident

#108

I wouldn’t want to be on the wrong side of a lawsuit, defending drunk employees working on production data. What outrageous recklessness. And how imprudent to admit this to the public. Some things are best kept to yourself. No one needs to know that.

thank you for your highly valued expert opinion.

edit: a stunning new record of bots flagging my post within 3 minutes. woo hoo...

never change hn.

Re: We deleted the production database by accident

#109
post #30

> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults

I feel that computers make it easier for this danger to be more indirect, however. The examples you give are physical, and even the youngest of child would likely recognise they are not regular items. A production database, meanwhile, is visually identical to a test database, if measures are not made to make it distinct. Adults though we may be, we're human, and humans can make really daft mistakes without the right…

So, silly idea. What if, to work on the production database, you had to go into the special production room, colored in the special production color, scented with the special production perfume, and sit on a just tiny bit uncomfortable production chair.

Basically make it clear even to the caveman brain that things are different.

Re: We deleted the production database by accident

#110
This is bad operations.

That it happened meant that there were many things wrong with the architecture, and summing up the problem to “these things happen” is irresponsible, most importantly your response to a critical failure needs to be in the mindset of figuring out how you would have prevented the error without knowing it was going to happen and doing so in several redundant ways.

Fixing the specific bug does almost nothing for your future reliability.

Post reply on HN