Not me, but I had to wake up at 3:00am and fix it. Someone ran “User.destroy_all” in a production Rails console. This was a social network for a ‘well know sports agency’. Production console’s are a sharp tool. Be careful who you hand them to.
Ask HN: Best “I brought down production” story?
151–160 of 301 posts
Re: Ask HN: Best “I brought down production” story?
#152I locked GitHub’s user table in the middle of the workday by adding a column. Oops.
Re: Ask HN: Best “I brought down production” story?
#153Earlier quoted context omitted.
Wait, what? So the system was purposefully hindering everyone's productivity? Sounds more like a "I brought up production" story...
My guesstimated translation: some large number of those users were running compile jobs, which were running as background tasks instead of foreground tasks.
Yes, because the system administrator, in his great wisdom made the compiler a batch job, but left the default priority way too low, below all interactive task.
Which meant the users had to wait around, so the labs would always end up full of logged in users, frustrated that the system was so darn slow.
Re: Ask HN: Best “I brought down production” story?
#154I gave my boyfriend at the time root access to my personal server
In a sleepy haze he accidentally ran a chown -R him:him / && chmod -R 777 /
Everything basically imploded on itself. I had to boot from a LiveCD and slowly correct the chmod/chown of every file and folder, using my local machine as a permission reference structure
Re: Ask HN: Best “I brought down production” story?
#155I cofounded a start up which was a call canter using software I wrote to manage maintenance at small pad retail sites such as fast food restaurants. We had investors. One showed up, a couple, who were extremely angry due to the repayment of their investment. It was a small building, but handled hundreds of calls per day, with 12 call-center reps. The investors were loud. At the same time, our database seemed to have locked up, it was on a Pyramid Unix server running Sybase, my choice. So I needed to address the stall, at the same being attacked with irate, high volume screaming, at me. While typing at my terminal, I accidentally "drop table"'d. I bricked the database and after 15 minutes on the phone with Sybase realized we had no choice but to restore from backup. That's when I found out from the tech in charge of backups that he had missed the last day. We lost 2 days of data and had to run with paper and pencil until we had the database up and working.
So we restored what we had, and over the next 5 days, as people called in for status etc., we reconstructed most of what we had lost. The reps worked with the callers, and we worked things out over a week of 24-hour days.
Embarrassing, but we lost zero customers. Unfortunately there were hundreds of small mishaps, and many unhappy people, but ultimately the business held on, went forward, and a few months later there was no lingering effect.
Lesson: Don't let investors visit where the business is done, and verify your backups by restoring them. I recommend avoiding situations like this, but if one happens you can work it out with great effort, and it is rarely the end of the world.
Re: Ask HN: Best “I brought down production” story?
#156Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…
`ssh $host /bin/sh` (or another shell) should work?
Depending on the distribution, /bin/sh might be a symlink to bash
Re: Ask HN: Best “I brought down production” story?
#157A friend of mine ran a large and relatively popular (as in at least 30 users online at any given time ...) PvP MUD on a server of mine back in the (late?) 90s. I didn't play muds and my experience was mostly limited to helping him fix C programming bugs from time to time and fielding an occasional irate phone call from users who got my number off the whois data. But because of the programming help I had some kind of…
Oh man. MUD's in the late 90s. Just randomly reminded me of all the time I spent in Medivia (I'm fairly sure that was the one). Those were good times. Surprised the name even came to me since I haven't thought about those in... 20ish years.
Medivia? Judging from screenshots that looks a lot like Tibia (https://www.tibia.com)
Re: Ask HN: Best “I brought down production” story?
#158I didn't do this personally but one enterprise I worked at ran their production DNS off an old sun ray desktop sitting on a random desk in the computer room. Eventually somebody who didn't think it was being used for anything just yanked the plug.
Re: Ask HN: Best “I brought down production” story?
#159As a new junior, remote, part-time hire, I got frustrated that my test suite wasn't passing on my laptop. So I checked out the codebase on a production server and ran my tests there.
The test suite initialized by truncating all tables and loading fixtures instead.
Using production database credentials.
Oops.
Re: Ask HN: Best “I brought down production” story?
#160Partition with the mSQL database on filled up; I moved it to /tmp. On a Solaris box. Which rebooted some weeks later.
Can't tell you how many weblogic "won't starts" I fixed over the years caused by people staging ear's and war's in /tmp on Solaris.