Ask HN: What's the worst you've ever screwed up at work?

Last day of work before moving to the new job: I do some cleanup and rm -fr my home directory. Seconds passed. Minutes passed. I start to think about how can it take so long.

I list the content of my home directory trying to understand which folder was so big. Then I see it. A folder usually empty. Empty because I use it as generic mount point. A mount point that the day before was attached via sshfs to the production server...

I had a strange feeling, like if I was seeing myself from behind, something crumbling inside me. And at that moment someone start to ask "what's happened to "?

I take my courage and I say "I know it"...

That was really hard. The worst day at work in years, and during the last day too. Luckily we had a good enough backup strategy and the damage was mostly solved in a couple hours.

There I realized how much of an idiot I was to have mounted the production server on my home and I grow a little.

I work at a newspaper as a programmer for the website. Mostly my job is backend programming, some HTML and CSS work (mostly left to designers). I run our local computer infrastructure as well as manage a cluster for our online presence and assist in technology related journalism as well as assisting our CEO in managing the IT budget.

I inherited a mess of an architecture and am finally getting around to rewriting our deployment process. We buy VM services from a local outfit and the prices are basically an arm and a leg for rather small machines. Due to this my predecessor put in place an insane deployment script. It pulls the new version from github then reloads code on the running dynos, one after another. Reverting is out of the question with our current approach to VCS (something I am also fixing). Most of the time this is no problem, all we are changing really is some template code, or introducing new models and their views.

Thinking back I am quite happy we don't run into more problems than we do, but also happy that this type of insanity is soon in the rearview mirror.

The worst mistake was recently, cost us about 4 hours of downtime during the busiest time of the day.

A big feature on all news sites are lists of stories to present to the user to look at after they have read what you put in front of them at the moment. They may take the form of most viral, most read, most commented, sliced by time or category or many other factors. My predecessor had written all those lists statically, which made maintenance a nightmare and extension very fragile.

I made a function that was a generic list of items. You supply basic parameters, amongst them a QuerySet for what would construct the list and my function would check to see if it was cached and if it wasn't, generate it and cache it.

The framework I use (Django) generally uses lazy evaluation for all QuerySets and I rarely have to think about the size of the list I generate, I just take care to limit the query before I list() it. During development nothing showed up as a problem and I deployed this and all seemed to be good with the world.

A week passes by where I made at least 2 minor deploys (small changes to templates, minor tweaks to list filters) and all seemed to be good with the world.

Designer sends me a pull request, I look over the code, just some garden-variety template changes, nothing that should raise an eyebrow. Make the merge, plan to deploy and then go to lunch. Deployment done, all seems well for 2 minutes but then suddenly servers lit on fire. Pages spewed 404's and 500's like there was no tomorrow.

For 4 hours I tear my hair out, examine every piece of code I was deploying that day, call in the big gun support (the kind that costs more money than I care to think about). Everything I was looking at pointed to the caching agent not working. Too many pageviews requesting the database, too much load on the servers, reboots made them work fine for about a minute but then everything became bogged down.

The big gun support pointed something out finally that I had missed: Traffic from the database to the dynos was abnormally high. Made me take a look at code that had been there for a while and lo and behold: For some reason when you pass a QuerySet as a parameter, it seems to be evaluated for the receiving function! 2 lines of code added, one deploy, problem fixed.

I have no idea to this day how this code could be live for a week without causing problems but an unrelated change triggers the bad behavior. This is not be the first time I've seen strange behavior from code, having seen a Heisenbug in Java code.

There's a happy ending to this. I made a big mea culpa slideshow where I pointed out all the flaws and what we needed to do to prevent a re-occurence. I got support to make the changes needed and my new cluster goes live day after tomorrow. Now I can carefully change NEW dynos for a deployment, keeping the old one's around if the shit hits the fan. I got some changes instituted in how we approach VC, something that's hampered work for a while. And we save money in the long run because we will no longer be paying an arm and a leg for the VMs (AND I got to learn about clustering machines with HA, goodstuff with gravy).

Not the worst, but certainly most infamous thing I've done: I was testing a condition in a frontend template which, if met, left a comment in the header HTML of all the sites we served. Unfortunately the condition was always met and I pushed the change without thinking. This was back in the day when bandwidth was precious and extraneous HTML was seriously frowned upon. We didn't realize it was in production for a week, at which point several engineers actually decided to leave it in as a joke. Then someone higher up found out and browbeat me into removing it, citing bandwidth and disk space costs.

Now, if you go to a CNET site and view source, there's a comment. I like to think of that as an homage to my original fuckup.

Actually, the customers get pissed off even when they get something for free, because it's just another sign of how incompetent we are :(

system("tar --delete-files -czf archive.tar.gz $datadir/");

What could possibly go wrong?