Live data from Hacker News

Accidentally destroyed production database on first day of a job

reddit.com

21–30 of 31 posts

Re: Accidentally destroyed production database on first day of a job

#23
post #15

Lots of red flags there: - Dev installation guides with credentials to prod. - Tests that delete everything, not just what they create. - The obvious of giving access to delete production on the first day. - Where they lacking backups?

Seems from the post that there were backups but they had never actually tested restoring from them.

If you have backups, but have not recently tested restoration, then you do not have backups.

Test your restoration process people!

Re: Accidentally destroyed production database on first day of a job

#24
post #8

For us, one of our junior Devs managed to wipe out all environments, all datacenters for one of our Elastic instances. They got handed a task to modify an index. The Dev Lead and Sr. Devs were 'too busy' and she stack overflowed how to do it and it was rubber stamped. What she did worked - but it wipes out all the existing documents when she dropped and recreated the indexes. Issue was it was a large enough collectio…

Nice story. Your dev lead/senior engineer should have never been "too busy" for letting someone do that on a first day unsupervised

Never too busy to fix a fuck-up that could have been prevented.

Re: Accidentally destroyed production database on first day of a job

#25
This is probably made up, but if real, the CTO should have fired himself. Some of the shittest practices I've ever heard of. Why would you use a real persistent database for your test scripts, whether it's a production instance or not? Spin up a lightweight mock server on the fly and autogenerate fake credentials. Heck, one of the test cases should be that you can't just delete the whole thing. Why on earth would you store credentials in an onboarding guide?

Anyway, for a real story, back in 2002 Sears before they totally tanked tried to open a home decor business called The Great Indoors. I was one of their first retail employees hired to work the stock room at a new opening. First day on the job, a week before the place was going to open, someone spends about 30 seconds showing me how to operate some forklift-like crate carrier called a wave or something like that, and I'm supposed to move some stuff to another floor via freight elevator. I accidentally accelerate when trying to slow down and promptly destroy the elevator.

Store manager was irate and promptly spends 10 minutes screaming at me in front of everyone and then sends me home permanently. You know what? 21 year-old me internalized that shit and believed I was actually at fault, but in retrospect, that place was bullshit and both Sears and The Great Indoors deserved the fate of eventually going out of business. The ensuring two decades have been up and down, but I'm in a great place now. I hope, if this really happened, that this junior dev landed all right, too. Life is way too short to stick with a toxic workplace, and when you don't have a family to feed and still have the freedom to just walk away, you absolutely should.

Re: Accidentally destroyed production database on first day of a job

#26

This is probably made up, but if real, the CTO should have fired himself. Some of the shittest practices I've ever heard of. Why would you use a real persistent database for your test scripts, whether it's a production instance or not? Spin up a lightweight mock server on the fly and autogenerate fake credentials. Heck, one of the test cases should be that you can't just delete the whole thing. Why on earth would you…

[deleted]

Re: Accidentally destroyed production database on first day of a job

#27
I am technical lead for one a collection of risk systems and trade storages for one of largest banks in the world.

We have simple rules to prevent significant data loss:

1. Deleting data is not permitted and mutable objects are highly discouraged. No deletes/modifications == greatly reduced possibility of data loss from app code errors.

I lied a little bit.

There are functions that can remove data, but they are hardwired to refuse to work unless it is beyond question that they can't remove live prod data.

For example, the function will refuse if the collection is not prefixed with "tmp" or "test.

For production objects we have a system where we basically do CoW and create new versions of the documents and vacuuming system that archives old versions. The archive is preserved for a minimum period of time (2 weeks) to give chance to react in case somebody makes a blunder and screws up some rules to remove too much.

2. Remove write access to the database from every single person. No single employee should have write access to the database. No single employee should bear responsibility of having to work with an account that can let them destroy the database. No employee has access to PROD credentials, which are generated automatically and not present in configuration or application server.

Instead, if you have a need to introduces some changes to the database, an app was created where you can write your code as a kind of job that can modify the database. The job is not allowed to take any parameters (only has a name) so that it is possible to audit what it is going to do exactly. This code then goes through regular development process including code reviews, automated tests, etc. Once deployed to PROD you can go to API and execute the job by name within PROD context.

Data loss prevention is a significant issue that no CTO should "dump" on his/her employees.

Also not accepting responsibility for everything that happens in their department is a sign of lack of leadership.

Re: Accidentally destroyed production database on first day of a job

#28

This is probably made up, but if real, the CTO should have fired himself. Some of the shittest practices I've ever heard of. Why would you use a real persistent database for your test scripts, whether it's a production instance or not? Spin up a lightweight mock server on the fly and autogenerate fake credentials. Heck, one of the test cases should be that you can't just delete the whole thing. Why on earth would you…

USA doesn’t require forklift licences?

Re: Accidentally destroyed production database on first day of a job

#29

This is probably made up, but if real, the CTO should have fired himself. Some of the shittest practices I've ever heard of. Why would you use a real persistent database for your test scripts, whether it's a production instance or not? Spin up a lightweight mock server on the fly and autogenerate fake credentials. Heck, one of the test cases should be that you can't just delete the whole thing. Why on earth would you…

USA doesn’t require forklift licences?

Does but I'm guessing this vehicle somehow qualified under regulatory minutiae as not actually a forklift.
Post reply on HN