Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

31–40 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#31
This was 20 years ago now - it was my first day in a new job working for a startup.

Our startup was based in the garden office of a large house and the production server was situated in a cupboard in the same room.

The day I started was a cold January day and I’d had to cycle through flooded pathways to get to work that morning - so by the time I arrived my feet were soaked.

Once I’d settled down to a desk I asked if I could plug a heater in to dry my shoes. As we were in a garden office every socket was an extension cable so I plugged the heater in to the one under my desk.

A few minutes later I noticed that I couldn’t access the live site I’d been looking through - and others were noticing the same.

It turned out the heater I was using had popped the fuse on the socket. The extension I was using was plugged into the UPS used by the servers. So the battery had warmed my feet for a few minutes before shutting down and taking the servers down too.

And that’s how I brought production down within 3 hours of starting my first job in the web industry…

Re: Ask HN: Best “I brought down production” story?

#33

Years ago, my employer was light on funds so we cobbled together plugs to use as a loopback when testing and identifying network jacks. It worked great. Insert the plugs in cubilces then test the open ports in the wiring closet. Worked great many times until one day we plugged it in and went to lunch. When we came back, we were told the network had slowed to a crawl and captures showed floods. This was during the day…

Exact same scenario but it was connected to the on premise data center. I was imaging devices and figured why not use a switch and provision 4 at a time. Started seeing everything go down figured the network team was doing things. Checked my images and they had stopped, unplug a cord and everything is working. I don't think anything of it and plug it back in and went back to waiting for the images to deploy but the storm started. After about 5 hours customers started receiving their electricity after the network team found my device.

Re: Ask HN: Best “I brought down production” story?

#36

Near miss: my first job I was working on a CRUD app for a huge bank. I was dumb and it was early in the enterprise era of software and I had built my own simple O/R tool based on codegen. Not a terrible tool all things considered and I was pretty pleased with myself. One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database. Not…

And this is why we use things like database users that don't have delete permission, and row-level security so users can delete things that don't belong to them.

I have learned this from a very similar experience.

Re: Ask HN: Best “I brought down production” story?

#37
post #35

We pushed a CDN config that triggered a CDN provider bug. Took down the CDN's entire presence on one continent. Broke a whole bunch of recognizable sites for a bit.

Are you referring to the Fastly outage that happened a few weeks ago, or is this more common than I realize?

Re: Ask HN: Best “I brought down production” story?

#38

Many years ago (decades, in fact) as a fresh new excuse for a unix admin, I needed to hide the passwd binary so that users couldn't find it and change their local password on the terminal box (this was early ISP days). SunOS 4.1.4, as I recall. Anyway, I hid that binary. In /etc, where they'd never think to look. Gosh we do some dumb things, eh? LOL. That took a while to find a solution for, and no small amount of lu…

I don't get it - how would moving that binary break a running system? Is that binary somehow involved in something else beyond password changes?

Re: Ask HN: Best “I brought down production” story?

#40

Many years ago (decades, in fact) as a fresh new excuse for a unix admin, I needed to hide the passwd binary so that users couldn't find it and change their local password on the terminal box (this was early ISP days). SunOS 4.1.4, as I recall. Anyway, I hid that binary. In /etc, where they'd never think to look. Gosh we do some dumb things, eh? LOL. That took a while to find a solution for, and no small amount of lu…

I don't get it - how would moving that binary break a running system? Is that binary somehow involved in something else beyond password changes?

/etc/passwd contains the user database on most Un*x systems. GP replaced it with the executable file, thus wiping out the system's users. Ouch.
Post reply on HN