Ask HN: What's the worst you've ever screwed up at work?

In the rm line of the snippet above, "/some/location". Magic variables in bash tend to lose their magic once set.

  # cd /etc
  # emacs inetd.conf
  # ls
  ...
  ... inetd.conf
  ... inetd.conf~
  ...
  # rm * ~
  # ls
  # ls

I had had some test tables sitting around in the database for a while and decided to clean them up. I stupidly forgot to check the status of my backups; because of an earlier error, they were not being correctly saved.

So, I had a bunch of tables with similar names:

    users_1024
    users_1025
    users_1026

Guess what got deleted along with them? The actual users table (which I've since renamed to something that does not even contain "users" in it).

So, how do you recover a users table when you've just deleted it and your backup has failed?

Well, I happened to have all of my users' email addresses stored in a separate mailing list table, but that table did not store their associated user IDs.

So I sent them all an email, prompting them to visit a password reset page.

When they visited the page, if their user ID was stored in a cookie -- and for most of them, it was -- I was able to re-associate their user ID with their email address, prompt them to select a new password, and essentially restore their account activity.

There was a small subset of users who did not have their user IDs stored in a cookie, though.

Here's how I tackled that problem:

Because the bulk of a user's activity on the site involves answering poll questions, I prompted them to select some poll questions that they had answered previously, and that they were certain they could answer again in the same way. I was then able to compare their answers to the list of previous responses and narrow down the possibilities. Once I had narrowed it down to a single user, I prompted them to answer a few more "challenge" questions from that user's history, to make sure that the match was correct. (Of course, that type of strategy would not work for a website where you have to be 100% sure, rather than, say, 98% sure, that you've matched the correct person to the account.)

So one of the first things I wanted to do was setup a development db for which I exported the structure from their prod db. I then proceeded to change the name of the create database statement at the top to the new dev db I wanted and ran the script.

Unfortunately the prod db name was still pretended to every drop and create table command in the script so I had just replaced their whole prod db with an empty one.

Owning up to that was one of the most embarrassing moments of my career. It was such a rookie mistake I just wanted to die. Luckily they had daily backups so I only cost their 4 man business about half a day of work but... it was enough for me to be a much more careful developer from that day forward!