We deleted the production database by accident

Post by cortesoft » Sat, Oct 17, 2020, 11:52 PM UTC

defen wrote: post #15

xnyan wrote: post #11

localhost is an abstraction, it's a non-routable-outside-your-machine network...except it's not. It's nothing more than normal TCP traffic except with a message to the OS and other programs that whatever is on that local computer network, you don't want it routed outside the local computer. There's absolutely nothing stopping anything with access to localhost from routing it anywhere that process wants. Does not even…

The fact that the production db has the same username/password as the development one is perhaps more troubling.

Post by at_a_remove » Sat, Oct 17, 2020, 11:53 PM UTC

amluto wrote: post #13

muststopmyths wrote: post #7

>Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. We’re too tired to figure it out right now. The gremlins won this time. Obviously, somehow the script ran on the database host. some practices I've followed in the past to keep this kind of thing from happening: * A script that deletes all the data can never be deployed to production. * scrip…

I have a little metadata table in production that has a field that says “this is a production database”. The delete-everything script reads that flag via a SQL query that will error out of it’s set in the same transaction as the deletion. To prevent the flag from getting cleared in production, the production software stack will refuse to run if the “production” flag is not set.

Post by mcpherrinm » Sat, Oct 17, 2020, 11:56 PM UTC

muststopmyths wrote: post #7

>Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. We’re too tired to figure it out right now. The gremlins won this time. Obviously, somehow the script ran on the database host. some practices I've followed in the past to keep this kind of thing from happening: * A script that deletes all the data can never be deployed to production. * scrip…

Post by defen » Sat, Oct 17, 2020, 11:57 PM UTC

cortesoft wrote: post #21

defen wrote: post #15

Earlier quoted context omitted.

The fact that the production db has the same username/password as the development one is perhaps more troubling.

It likely doesn't... it probably reads it from the environment or a config role, and since it was in production it had the production credentials.

Post by malwrar » Sat, Oct 17, 2020, 11:57 PM UTC

webel0 wrote: post #16

This is my greatest fear when it comes to terraform: > terraform destroy (And either a confirmation or a flag) and everything is deleted. I know you can add some locks but still :/