Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#231

Post by klysm »

GordonS wrote: post #218

In early 00's I got my first gig as a web developer for an oil company. They had no other developers. There was some kind of security incident, and the PHB (literally) wanted to beef up security. For some reason, infosec was a kind of hobby for me, and I suggested (and volunteered) to install Snort as a company-wide wire tap. So, during a maintenance period I installed the tap and server where the cable came in. And…

Yeah that’s really not supposed to happen - was one of the MACs redefined or was it a manufacturer error?

Re: Ask HN: Best “I brought down production” story?

#232

Post by secondcoming »

When I worked in adtech, we would load ads from a sqlite file. I rolled out a schema change but forgot to update the code which retrieved query data by column index rather than name.

We served a few million ads that contained nothing but the text "255". It was quite expensive.

Re: Ask HN: Best “I brought down production” story?

#233

Post by Nextgrid »

krylon wrote: post #40
Nextgrid wrote: post #38

Earlier quoted context omitted.

I don't get it - how would moving that binary break a running system? Is that binary somehow involved in something else beyond password changes?

/etc/passwd contains the user database on most Un*x systems. GP replaced it with the executable file, thus wiping out the system's users. Ouch.

My bad. I somehow expected him to put it into /etc and rename it to something else. Indeed if he overwrote /etc/passwd then all hell would break loose.

Re: Ask HN: Best “I brought down production” story?

#234

Post by mijoharas »

linsomniac wrote: post #107
hamburglar wrote: post #62

Earlier quoted context omitted.

Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”

Probably not if it was overwritten (": >/bin/bash") rather than removed and recreated ("rm -f /bin/bash; : >/bin/bash"). The former will cause all processes to see the empty file, the latter would leave processes with access to the old contents. In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you…

I was unaware of the first part of this, and decided to find a reference.

This stack overflow answer[0] provides some detail on ETXTBUSY and the fact that writing directly writing to a running executable file will fail. (I had thought it would unlink the file like in the second case, simply because that's what I always see when recompiling something that's already running. This obviously uses the second method though).

[0] https://unix.stackexchange.com/questions/187931/modifying-bi...

Re: Ask HN: Best “I brought down production” story?

#235

Post by joddystreet »

joddystreet wrote: post #174

I was sitting in an escalation-call, called by one of our most important customer and India’s largest ISP. There were 13 of them, 3 of us, and only 1 technical (that’s me). Things were super intense, extremely heated debates were going on, I was fixing a few issues, while answering the questions coming my way, in parallel I was also chatting with my teammates, who were back in the office and were on support calls wit…

BTW, the software is still being used by their OPS team, it’s used to monitor a third of India’s internet backbone infrastructure.

The guys who caught me were the actual users of the software, their team actually.

Saves them a millions of $ per annum, because of being able to meet SLA.

It’s now running for 7 years in their NOC.

Users revolted against their CIO (second or third replacement CIO), who proposed replacing this software (and others in their NOC), with an HP unified monitoring suite(a part of CIO’s digital transformation initiative). Apart from this software, rest of the monitoring systems have been replaced.

:)

Re: Ask HN: Best “I brought down production” story?

#236

Post by samus »

avh02 wrote: post #229

When i was working for one of the biggest regional websites: We had blue/green deployments using AWS Elastic Beanstalk - our actual deployment process was still manual and we had an informal (i.e: not written down but everybody knew) checklist to follow - one of the steps involved checking that capacities matched. Well, everything looked good and I switched environments, as I was checking that the website was still u…

Not too bad. Unless you have a contract or regulations that says otherwise, 5 mins downtime once in a while is really nothing to lose your sleep about. Did you even have to report it?

Re: Ask HN: Best “I brought down production” story?

#237

Post by samus »

etaioinshrdlu wrote: post #6

I worked in online advertising and pushed infinite loops that froze browsers to millions of unsuspecting victims. On another occasion I had a division operation happen on integers instead of floats, and the code was running on some hardware that steered antennas for radios on airplanes. Much time was spent by pilots flying in circles over LA while I gathered data and found the "oops". It was fixed by adding a period…

> On another occasion my machine learning demo API failed due to heavy load, but only when India's prime minister was looking at it.

This really sounds like something that can only happen when unpredictable machine learning models are involved^^

Re: Ask HN: Best “I brought down production” story?

#239

Post by junon »

The long-running one at Uber for a while was that someone accidentally pasted a Spotify link into some JSON directly after a bracket or something, causing it to be invalid, and brought down the entire API layer after the deployment failed to detect any errors.

Personally, on my first day of my last job, I was brought on to improve the backend (turns out, only how the non-programmer CEO decided was okay, but that's beside the point). I slow-loris'd production just in the off-chance it was vulnerable, because no way would a site hosted on AWS by a tech lead that claimed he was competent would have a slow-loris-vulnerable setup.

It was indeed vulnerable, and not only did I bring down production, I also (somehow) brought down our own wifi router in the office. Caused a few hours of downtime on a monday.

Learned my lesson that day - even though everyone on the team says soandso is super elite, wait until they show you they are and don't trust that they did anything right.

Re: Ask HN: Best “I brought down production” story?

#240

Post by GordonS »

klysm wrote: post #231
GordonS wrote: post #218

In early 00's I got my first gig as a web developer for an oil company. They had no other developers. There was some kind of security incident, and the PHB (literally) wanted to beef up security. For some reason, infosec was a kind of hobby for me, and I suggested (and volunteered) to install Snort as a company-wide wire tap. So, during a maintenance period I installed the tap and server where the cable came in. And…

Yeah that’s really not supposed to happen - was one of the MACs redefined or was it a manufacturer error?

Manufacturer error!

IIRC, the NICs were the same make & model.

Post reply on HN