Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

41–50 of 456 posts

Re: We deleted the production database by accident

#43

Post by Guest42 »

junglejoose wrote: post #32

You aren’t a real engineer until you do this. So congrats on the promotion! :)

Definitely, the likelihood of these things happens goes way up alongside the amount of concurrent tasks, meetings, or other forms of distraction. I’ve seen my fair share of production administration as a side task.

Re: We deleted the production database by accident

#44

Post by codegladiator »

geofft wrote: post #35
cblconfederate wrote: post #28

> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults

I'm not sure I follow your point - I think you'll find the same attitude towards complexity by operators of military rockets and nuclear power plants. If you look at postmortems/root-cause analyses/accident reports from those fields, you'll generally find that a dozen things are going wrong at any given time, they're just not going wrong enough to cause problems.

We already know humans make mistakes. But for this particular scenario lets blame computers.

Re: We deleted the production database by accident

#46

Post by hannofcart »

It takes a great deal of integrity to admit that you deleted a database because you were mucking around in your infra after red wine.

And it bodes well for your firm that that doesn't get you fired either.

These things happen to the best of us but having dealt with it responsibly and honestly as a team is something you can be proud of IMO.

Re: We deleted the production database by accident

#47

Post by amingilani »

Happens to all of us. Once I required logs from the server. The log file was a few gigs and still in use. so I carefully duplicated it, grepped just the lines I needed into another file and downloaded the smaller file.

During this operation, the server ran out of memory—presumably because of all the files I'd created—and before I know it I'd managed to crash 3 services and corrupted the database—which was also on this host—on my first day. All while everyone else in the company was asleep :)

Over the next few hours, I brought the site back online by piecing commands together from the `.bash_history` file.

Re: We deleted the production database by accident

#48

Post by codegladiator »

> Computers are just too complex and there are days when the complexity gremlins win.

> However, we will figure out what went wrong and ensure that that particular error doesn’t happen again.

How can you say statement 2 just after statement 1 ? Isn't statement 1 just plain acceptance of defeat ?

And looking at all the replies here, is this a feel good thread for the mistakes you made ?

Re: We deleted the production database by accident

#49

Post by tluyben2 »

rullelito wrote: post #4

LOL, I forgot how few safety measure startups have in place.

Like this never happens in big corps: sure, mostly on departmental level but still enterprises, not startups. They try to escape the annoying and slow as molasses dba’s/devops and install/create/buy/saas systems to avoid the red tape. But then the same things as with startups go wrong. And rather often too; we often had these calls asking if we could restore.

Obviously there are plenty of large enterprise wide data breaches, which, I would say is actually worse than losing a day of data in a lot of cases. So also not so many satefy measures, again, worse than startups; at least they have an excuse of being understaffed and underfunded.

Post reply on HN