Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#121

Post by rrauenza »

chasd00 wrote: post #92
alfiedotwtf wrote: post #53

Earlier quoted context omitted.

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root. we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.

I was taught to also use a transaction and then check how many rows were affected before committing.

Re: Ask HN: Best “I brought down production” story?

#122

Post by lamontcg »

I was a system engineer at Amazon from 2001-2006. Sometime around 2004/2005 or so there was a development team working on the "a9 search engine" (meant to complete with google) down in SF. They were sort of an official "shadow IT" offshoot and asked for special treatment and they got me assigned specifically to them to build out the first of their two webservers.

They did the usual mistake of wanting to jettison all the developer tooling and start from scratch. So there was a special request to just install a base O/S, put accounts on the box, and setup a plain old apache webserver with a simple /var/www/index.html (this was well outside of how Amazon normally deployed webservers which was all customized apache builds and software deployment pipelines and had a completely different file system layout).

They didn't specify what was to go into the index.html files on the servers.

So I just put "FOO" in the index.html and validated that hitting them on port 80 produced "FOO".

Then I handed off the allocated IPs to the networking team to setup a single VIP in a loadbalancer that had these two behind it.

The network engineer brought up the VIP on a free public IP address, as asked.

What nobody know was that the IP had been a decomissioned IP for www.amazon.com from a year or two earlier when there was some great network renumbering project and it had pointing at a cluster of webservers on the old internal fabric.

The DNS loadbalancers were still configured for that IP address and they were still in the rotation for www.amazon.com. And all they did as a health check was pull GET / and look for 200s and then based on the speed of the returns they'd adjust their weighting.

They found that this VIP was incredibly well optimized for traffic and threw most all the new incoming requests over to these two webservers.

I learned of this when my officemate said "what is this sev1... users reporting 'foo' on the website..."

This is why you always keep it professional, kids...

Re: Ask HN: Best “I brought down production” story?

#123

Post by alfiedotwtf »

kryptk wrote: post #64
alfiedotwtf wrote: post #53

Earlier quoted context omitted.

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

When your database is too big to backup that's a lot like a bank being too big to fail. Unwise long term strategy.

Terabytes across 25+ databases not co-located and all being masters replicating to each other, in 2009. Not sure how you would have done it.

Re: Ask HN: Best “I brought down production” story?

#124

Post by ShakataGaNai »

nullc wrote: post #80

A friend of mine ran a large and relatively popular (as in at least 30 users online at any given time ...) PvP MUD on a server of mine back in the (late?) 90s. I didn't play muds and my experience was mostly limited to helping him fix C programming bugs from time to time and fielding an occasional irate phone call from users who got my number off the whois data. But because of the programming help I had some kind of…

Oh man. MUD's in the late 90s. Just randomly reminded me of all the time I spent in Medivia (I'm fairly sure that was the one). Those were good times. Surprised the name even came to me since I haven't thought about those in... 20ish years.

Re: Ask HN: Best “I brought down production” story?

#125

Post by lenova »

mansoon wrote: post #115
maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

It's a poetic way of saying the boss was/is a better person than the OP.

Re: Ask HN: Best “I brought down production” story?

#126

Post by oceanghost »

takoid wrote: post #75
oceanghost wrote: post #17

Can It be a story I was involved in but I didn't do it? I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves. Our user directories were about 100GB at the time. I had sourced this special tape drive…

Hilarious story, thanks for sharing. > "Is the light on the AIT array green or amber?" Can you explain this? What is an AIT array?

As the others have correctly intuited-- its a tape backup system. We couldn't afford a proper tape library with a robot. I can't remember why, but there was a limitation of the software or SunOS that we needed to be able to get an L0 onto one tape. This thing was two Sony AIT tape machines that had a special SCSI board that made them look like one single drive to the host thus doubling the capacity. It was just enough to skate by. I didn't have much faith in differentials and didn't like to let them run more then a few days as well.

I always assumed the fault was in that SCSI board. The hand-off between tape-1 and tape-2 was what usually failed. The problem might occur 24 hours into a backup so it was difficult to get good backups. Also, I was not a full time employee (being a student), so I couldn't babysit this thing 5 days a week like a full time employee. Also, it kind of killed the performance of the system, so I had to do them at odd hours.

I am proud to say I never lost a single bit at that job.

In essence if this backup had failed months of research would have been lost (maybe two to 4 week old backup x 12 researchers).

Anyways, thank you for reading my silly story!

Re: Ask HN: Best “I brought down production” story?

#127

Post by oceanghost »

mrleinad wrote: post #66
oceanghost wrote: post #17

Can It be a story I was involved in but I didn't do it? I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves. Our user directories were about 100GB at the time. I had sourced this special tape drive…

I take it deleting everything was an accident, right? Not that the guy applied for a job only to destroy that information... lol

I do not actually know what happened. But something basically resulted in an rm -rf /home and was accidental.

By all reports, he eventually became a well liked and good admin. I just don't think he knew that much Unix when he started.

Re: Ask HN: Best “I brought down production” story?

#128

Post by RHSeeger »

sneak wrote: post #85
nullc wrote: post #80

A friend of mine ran a large and relatively popular (as in at least 30 users online at any given time ...) PvP MUD on a server of mine back in the (late?) 90s. I didn't play muds and my experience was mostly limited to helping him fix C programming bugs from time to time and fielding an occasional irate phone call from users who got my number off the whois data. But because of the programming help I had some kind of…

See also: https://en.wikipedia.org/wiki/Corrupted_Blood_incident

That was an annoying time in the game. Not horrible, but you pretty much had to avoid town because people were purposefully trying to spread the plague. Certainly memorable though.

Re: Ask HN: Best “I brought down production” story?

#129

Post by dotancohen »

jhugo wrote: post #65
theon144 wrote: post #14

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…

`ssh $host /bin/sh` (or another shell) should work?

Logging into another shell would be attempted only if someone knew at the time why the logins were failing.

But thanks, I've just added another technique to my toolbox.

Re: Ask HN: Best “I brought down production” story?

#130

Post by quacked »

notatoad wrote: post #116
mansoon wrote: post #115

Earlier quoted context omitted.

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

It's not good to punish people for making mistakes in the course of their work (especially if that work is meant to be educational) It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access. Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the cons…

Wait, but if you choose the guy giving access as the mistake made in the course of his work, then what's the new plan?
Post reply on HN