We've all been there (most of us, at least). What did you do (or not do), how did you first react, and how did you handle it?
Bonus points for sharing what you learned/key takeaways from the experience.
Ask HN: What's the worst you've ever screwed up at work?
1–10 of 322 posts
Re: Ask HN: What's the worst you've ever screwed up at work?
#2Many years ago, when I was but a fresh faced idiot, the partition that contained the mSQL database which had All The Data filled up. I moved it into /tmp because there was plenty of space.
On a Solaris box.
Hilarity ensued when we next rebooted it.
Re: Ask HN: What's the worst you've ever screwed up at work?
#3About 30 years ago I deleted the JOBCONTROL process on an old VAX 11/780 thinking it might be the reason why someone's process was stuck.
It wasn't a but an hour before I lost sysadmin privileges.
Never "experiment" with a production system - ever.
Re: Ask HN: What's the worst you've ever screwed up at work?
#4The worst thing I have done is terminate a running production instance with no database backups.
Client, not happy.
Re: Ask HN: What's the worst you've ever screwed up at work?
#5In late 2008 when I was in the Marines and deployed to Iraq I was following too closely behind the vehicle in front while crossing a wadi and we hit an IED (the first of 3 that day).
Nobody was killed, but we had a few injured. Thankfully the brunt of it hit the MRAP in front of us. If it hit my vehicle (HMMWV, flat bottom) instead I probably wouldn't be here.
That was the first major operation on my first deployment, too. Hello, world!
My takeaway? Shit just got real.
We ended up stranded that night after the 3rd IED strike (our "rescuers" said it was too dangerous to get us). It was the scariest day of my life, but in similar future situations it was different. I still felt fear and the reality of the existential threat, but I accepted it. It was almost liberating. Strange.
I deployed for another year after that (to Afghanistan that time). After Afghanistan I left the Corps and started my company. Because if it fails, what's the worst that can happen? Lulz.
Re: Ask HN: What's the worst you've ever screwed up at work?
#6First job, circa 2000, at an ISP that was run very clearly as a business and cutting corners. Not only was it critically understaffed, but management was more interested in laughing their way to the bank than management. They had me - with literally no routing protocol experience - manage a live route advertisement transition between two peering providers. Result: all customers offline, ~24 hours.
Reaction was standard: mostly to point out I did my best in unfamiliar territory and things should be sorted soon.
Take aways were: (1) less support calls than expected - users put up with things. (2) you learn when you fail (3) always have a backup
They kept me on at that job but I left pretty soon anyway as I got a 'real' (as in creative) job hacking perl-powered VPN modules for those Cobalt Raq/Qube devices, and building a Linux-related online retail venture for the same employer ... that worked great, but failed commercially.
Re: Ask HN: What's the worst you've ever screwed up at work?
#7In 2001 my first IT tech job as help desk analyst I heard beeping in the server room on one of the Solaris/Oracle machines and pressed the power off/power on button on the chassis. DBA came running in and I promptly left saying "oh I think it rebooted itself". The company went bankrupt shortly after so no huge lashing came my way but all my more experienced friends where like "wtf never do that again!"
Re: Ask HN: What's the worst you've ever screwed up at work?
#8Introducing a master/minion update system to work I ran a batch update to take a certain percentage out of the cluster.
Unfortunately I got my selection criteria wrong and pulled out all of one cluster and half of a second, halting a few thousand operations.
Luckily the monitoring system was very quick to alert me of this and using the same (wrong) selection criteria it was a fairly simple process to stop the update and put them all back in the cluster.
Takeaways? The age old cliche of "With great power comes great responsibility". Oh and have good monitoring!
Re: Ask HN: What's the worst you've ever screwed up at work?
#9In late 2008 when I was in the Marines and deployed to Iraq I was following too closely behind the vehicle in front while crossing a wadi and we hit an IED (the first of 3 that day). Nobody was killed, but we had a few injured. Thankfully the brunt of it hit the MRAP in front of us. If it hit my vehicle (HMMWV, flat bottom) instead I probably wouldn't be here. That was the first major operation on my first deployment…
This really puts some of the boneheaded moves I've made in my career in perspective. One thing that's always kept me pretty even keeled after a blowup is to take a breath and tell myself that no matter how bad I've screwed up, I'm still here, still breathing, and there (most likely) is some way out of the hole I've dug, no matter how painful.
Depending on the industry, that might not be the case though. Thanks for your service.
Re: Ask HN: What's the worst you've ever screwed up at work?
#10First job, circa 2000, at an ISP that was run very clearly as a business and cutting corners. Not only was it critically understaffed, but management was more interested in laughing their way to the bank than management. They had me - with literally no routing protocol experience - manage a live route advertisement transition between two peering providers. Result: all customers offline, ~24 hours. Reaction was standa…
I worked at an ISP in NY exactly around 1997-2004 we had also the Rac/Qube devices and I had to manage stuff I was not familiair with :) I learned so much by trial by fire.