What is your best "and then I brought down production" story?
Ask HN: Best “I brought down production” story?
1–10 of 301 posts
Re: Ask HN: Best “I brought down production” story?
#2I had a product that had mostly stopped growing in usage. It was running on say fifty machines. I had put considerable effort into some memory optimizations, which was the scaling point for new hardware, so I talked Ops into bunching the active traffic load onto fewer machines. All of the active traffic load. Started hitting the memory limits (32 bits linux) and our server framework exited on malloc failure, so lots of exiting of long lived processes and loss of expensive state, delayed alerts etc.
I still think it was over-provisioned, but they told Ops to stop listening to me unless someone else agreed. Probably ran on the 50 machines till it was discontinued 10 or 15 years later, but I left so who knows.
Re: Ask HN: Best “I brought down production” story?
#3Years ago, my employer was light on funds so we cobbled together plugs to use as a loopback when testing and identifying network jacks. It worked great. Insert the plugs in cubilces then test the open ports in the wiring closet. Worked great many times until one day we plugged it in and went to lunch. When we came back, we were told the network had slowed to a crawl and captures showed floods. This was during the days of primitive DoS via broadcast floods. Well, this flood was self induced. The loopback plug was inserted into a jack that had a connection back to the network hub. It dutifully retransmitted everything it saw back onto the network. Whoops.
Re: Ask HN: Best “I brought down production” story?
#4First time I got into the computer room back in the days when one unix mini-computer ran hundreds of terminals. I was asked to put a reel tape into the drive to load a tar archive into the Oracle database.
I couldn’t get the tape drive door open so I looked around and saw a key next to the door.
That didn’t open the door either. I was stood there scratching my head when the double doors burst open and half a dozen sysadmins came running in like a SWAT team.
I was a bit surprised until I glanced down and notice all the lights were off.
Yes the key that turned the power off had been left in the machine.
Re: Ask HN: Best “I brought down production” story?
#5I had web server access, but they wouldn't give me a DB login.
So I crafted an .asp to do my maintenance.
Only I was calling CreateObject() in the for loop to get a new AdoDb.Connection for each of the array entries of the data.
That creaky IIS server crashed like the economy.
Re: Ask HN: Best “I brought down production” story?
#6I worked in online advertising and pushed infinite loops that froze browsers to millions of unsuspecting victims.
On another occasion I had a division operation happen on integers instead of floats, and the code was running on some hardware that steered antennas for radios on airplanes. Much time was spent by pilots flying in circles over LA while I gathered data and found the "oops". It was fixed by adding a period to an int literal.
On another occasion my machine learning demo API failed due to heavy load, but only when India's prime minister was looking at it.
Re: Ask HN: Best “I brought down production” story?
#7I was 18 and had just taken over the website of a car and horse trailer dealership. I typed “;rm -fR ~” into an e-mail form to show my coworkers what would happen. On purpose. I quickly restored it (I was ready for this.) They were pretty amused. We had no concept of “on prod”. “Dev” was my local PC. Damn teenager.
Re: Ask HN: Best “I brought down production” story?
#8Doom on a single PC was fun. But the same game on our network with all those broadcasts running through our net, which hosted a university's PCs. That was bad. And I started it several times until I noticed that "I" was the cause for network slowdown.
Re: Ask HN: Best “I brought down production” story?
#9Near miss: my first job I was working on a CRUD app for a huge bank. I was dumb and it was early in the enterprise era of software and I had built my own simple O/R tool based on codegen. Not a terrible tool all things considered and I was pretty pleased with myself.
One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database.
Not a fun drive in.
Yes, we drove in in those days.
Re: Ask HN: Best “I brought down production” story?
#10Very first job, as an intern, I was tasked with building a "free text search engine" for the product, using their api. Maybe my first week or so there, I left a script running over lunch. Turns out the internal IP addresses weren't subject to the rate limiting, and my script's queries were growing exponentially (I was sending the response back to the same endpoint which was querying with the response, and giving me back a larger response etc..) Within 20-30 minutes or so every production machine was stuck running one of my queries. And it happened on the day that the engineering team were taking the new intern out for a team lunch...
At the time I was mortified, but in hindsight the fact that I was able to do that in the first place was really the issue, not my script.