Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#91

Post by cvrjk »

Sent in a SQL command deleting multiple million records (intended) wrapped in a single transaction (not intended). The replication queues could not keep up and failed, bringing down most of the replicas. Master server kept trying to recover and maxed out all connections - no DBA could log in to perform manual recovery. We had to hard reboot not knowing what state the system is in and how long it'll take to fully recover. Did I mention that this was a few hours before trading was going to begin?

TBH, my team was very gracious about it and the RCA focused purely on the events that occurred and how to never let if happen it again. No blame game at all.

Re: Ask HN: Best “I brought down production” story?

#92

Post by chasd00 »

alfiedotwtf wrote: post #53
breakfastduck wrote: post #23

Earlier quoted context omitted.

Lack of validation? I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root.

we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.

Re: Ask HN: Best “I brought down production” story?

#94

Post by lights0123 »

bentcorner wrote: post #69
theon144 wrote: post #14

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…

Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.

Recovery disk, then either copy the disk's copy of bash (if it doesn't depend on a later glibc version), copy another shell to /bin/bash (as the system probably doesn't depend on bash-specific commands to boot), chroot and use the package manager, or use the package manager with an explicit sysroot (e.g. pacman --sysroot). The first two steps are very easy compared to the latter two, but should be followed by a reinstallation of the package that provides bash.

Re: Ask HN: Best “I brought down production” story?

#95

Post by bombcar »

ackbar03 wrote: post #67

I'm a guy in two person startup team, mostly handle the technical stuff. Woke up to see some error on our main work machine, which also hosts some of the services, after a sudo apt update. Wasn't the first time and was usually able to fix by just googling what the error is as on SO and running it. Did it again, told me to uninstall nvidia drivers, proceeded to do so, and bricked my hard disk. Was completely gut wrenc…

Could have been coincidental- the drive was about to die and that triggered it.

Re: Ask HN: Best “I brought down production” story?

#96

Post by justin66 »

haolez wrote: post #73
nullc wrote: post #68

On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…

Genuine question: what's the usefulness of Solaris' default behavior? Why kill all processes?

It's used in the Solaris init scripts for shutdown, shortly before the system tries to unmount all mounted disk volumes.

I made the same mistake nullc made once, as I was more accustomed to linux than solaris. That was after hours and the effect was immediate, but it was still a pretty jarring and memorable moment.

Re: Ask HN: Best “I brought down production” story?

#97

Post by monkeydreams »

cvrjk wrote: post #91

Sent in a SQL command deleting multiple million records (intended) wrapped in a single transaction (not intended). The replication queues could not keep up and failed, bringing down most of the replicas. Master server kept trying to recover and maxed out all connections - no DBA could log in to perform manual recovery. We had to hard reboot not knowing what state the system is in and how long it'll take to fully reco…

> TBH, my team was very gracious about it and the RCA focused purely on the events that occurred and how to never let if happen it again. No blame game at all.

Which is how a PIR, PER or PCR should be. If you don't understand why someone makes a mistake, you can't avoid future mistakes.

Re: Ask HN: Best “I brought down production” story?

#98

Post by mikewarot »

mrleinad wrote: post #77
mikewarot wrote: post #56

It was 1985, I was in a VAX computer lab with about 40 other people typing on the VT100 terminals... I ran a program to compute something, and forgot that I had bumped my priority wayyy up... everything in the room stopped, even the line printer, everyone went ohhhhhh. 10 seconds later, my program finished... and everything snapped back to life. Another time, I walked into a different, bigger lab, with 100 terminals.…

Wait, what? So the system was purposefully hindering everyone's productivity? Sounds more like a "I brought up production" story...

No, it was just a bad configuration choice by the system administrator.

Re: Ask HN: Best “I brought down production” story?

#99

Post by chasd00 »

pandemic_region wrote: post #22

The year is 2002, OS is Solaris, trying to compile some httpd add-on straight on the production server (because why not) kept giving some weird error about /etc/ld.so not being right. So junior me does: $ rm /etc/ld.so*

That one hurts

Re: Ask HN: Best “I brought down production” story?

#100

Post by nerdawson »

The deployment process for a site I used to work on would entail changing the owner of a folder. You would change the owner from the web server to your user, upload files, and then change it back.

sudo chown -R www-data:www-data [folder]

I’d made some changes and was ready to update the owner only I was inside the folder that needed updating. In the moment I decided the correct way to refer to that folder was /

I noticed the command was taking far longer than usual to execute. I realised the mistake but by then the server was down with no way to bring it back up.

Post reply on HN