Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#61

Post by mickotron »

Had a script that managed disk space on windows servers. The config file was xml, and Powershell did not treat the commented child element as nothing, it imported a blank child element in the PSObject.

So when it ran it looked in blank directory (default c:\windows\system32), for blank conditions. Zip filetype matching blank. Delete files over blank age.

These servers were rebuilt over a weekend, and the script was scheduled again, and broke these servers again, requiring another rebuild.

When I came in on Monday, I was told that the script had caused this carnage. I didn't believe them until I read the debug logs in horror.

Luckily the config was specific yo a subset of servers, but they happened to be servers that were responsible for police GPS radios to function.

Suffice to say it now has a lot of defensive programming in it to test the config file and resulting config object before doing anything.

Re: Ask HN: Best “I brought down production” story?

#62

Post by hamburglar »

theon144 wrote: post #14

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…

Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”

Re: Ask HN: Best “I brought down production” story?

#63

Post by hackeraccount »

Server running bind with multiple people jumping in and making edits to the config. A body goes in and makes an edit but never reloads the service. After that change I went in with my own change - my change was very minor and I knew it was correct so like a fool I didn't run a syntax check - and then I reloaded the service. I didn't even check after the case to make sure it was still running.

Narrator: Bind was not running.

Down goes a media organizations web site.

Re: Ask HN: Best “I brought down production” story?

#64

Post by kryptk »

alfiedotwtf wrote: post #53
breakfastduck wrote: post #23

Earlier quoted context omitted.

Lack of validation? I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

When your database is too big to backup that's a lot like a bank being too big to fail. Unwise long term strategy.

Re: Ask HN: Best “I brought down production” story?

#65

Post by jhugo »

theon144 wrote: post #14

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…

`ssh $host /bin/sh` (or another shell) should work?

Re: Ask HN: Best “I brought down production” story?

#66

Post by mrleinad »

oceanghost wrote: post #17

Can It be a story I was involved in but I didn't do it? I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves. Our user directories were about 100GB at the time. I had sourced this special tape drive…

I take it deleting everything was an accident, right? Not that the guy applied for a job only to destroy that information... lol

Re: Ask HN: Best “I brought down production” story?

#67

Post by ackbar03 »

I'm a guy in two person startup team, mostly handle the technical stuff. Woke up to see some error on our main work machine, which also hosts some of the services, after a sudo apt update. Wasn't the first time and was usually able to fix by just googling what the error is as on SO and running it. Did it again, told me to uninstall nvidia drivers, proceeded to do so, and bricked my hard disk. Was completely gut wrenching. Although most of the important stuff was backed up on repos, still had to rebuild the damn thing. Still not sure what happened exactly till this day

Re: Ask HN: Best “I brought down production” story?

#68

Post by nullc »

On Linux killall lets you killall all processes matching a name.

On Solaris killall kills all processes.

To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down.

Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate. "So why did you kill all the processes?" "Because I didn't expect it to do that." "But the command name is kill all?" ...

Re: Ask HN: Best “I brought down production” story?

#69

Post by bentcorner »

theon144 wrote: post #14

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…

Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.

Re: Ask HN: Best “I brought down production” story?

#70

Post by luke2m »

unethical_ban wrote: post #55
krylon wrote: post #40

Earlier quoted context omitted.

/etc/passwd contains the user database on most Un*x systems. GP replaced it with the executable file, thus wiping out the system's users. Ouch.

Why did you write it Un*x? Is there a Unex or Unox? I've seen it written *nix to grab Linux and Unix.

Linix?
Post reply on HN