Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#251

Post by rurban »

mrunkel wrote: post #241

Around 1999/2000 I worked at Universal Studios in Burbank. Shortly after I started working there in the networking group, a fellow employee asked me to reset a port on the core token ring switch. Being a helpful type, I clicked "disable" on the network management tool for the port and promptly everyone lost connectivity. To everything. A quick walk over to the NOC confirmed that yes, the entire campus was down, as we…

Nice. Reminds me of the night where I switched off 30% networking of an entire country. MySQL was involved. It didn't make the front page of the news though, which was our cutoff

Re: Ask HN: Best “I brought down production” story?

#252

Post by hackeraccount »

joddystreet wrote: post #174

I was sitting in an escalation-call, called by one of our most important customer and India’s largest ISP. There were 13 of them, 3 of us, and only 1 technical (that’s me). Things were super intense, extremely heated debates were going on, I was fixing a few issues, while answering the questions coming my way, in parallel I was also chatting with my teammates, who were back in the office and were on support calls wit…

"shell-shocked, couldn’t hear a thing, just sitting there frozen" Been there.

Re: Ask HN: Best “I brought down production” story?

#253

Post by samus »

aidenn0 wrote: post #109

I was testing backups for a CMS; to test, I had to destroy the test database. I destroyed the production database. While my manager was querying the CMS to do my annual review. Good news is I was already planning on restoring the test database from the production backup, so i had the database up in under 45 minutes (slower than it should have been because Oracle's docs were flat out wrong). A more senior engineer tol…

That honestly hurt to read. It's my worst nightmare to do something complicated and to accidentally mess up production data. Good thing you managed to restore it so quickly...

Re: Ask HN: Best “I brought down production” story?

#254

Post by yyyk »

truetraveller wrote: post #106
bombcar wrote: post #90

Earlier quoted context omitted.

And why it’s often best to mark a record deleted and then have a reaper remove the records at a later point. But you must make sure all normal queries don’t see deleted items.

That's over engineering. At this point, just rely on PITR. FWIW, postgres does have a "reaper" via vacuum, but not for the purpose of safety. but rather to allow for mvcc.

There are other advantages for soft-deletes, like not having to worry about FKs.

Re: Ask HN: Best “I brought down production” story?

#255

Post by shrike »

My first IT job was in a large call center. I was sweeping up in a data center and there was a keyboard cable stretched across a walkway. The keyboard wasn't movable (I don't remember why) so I unplugged it from the PC, swept around it, and plugged it back in. About 2 minutes later half a dozen people run into the room. Apparently the SUN workstation I had unplugged the keyboard from was a critical component of the call manager and there was a bug that forced a reboot when the keyboard was plugged in while the system was running. I had hung up on ~36,000 people. Lesson learned and I only had to keep the data center clean for another year.

Re: Ask HN: Best “I brought down production” story?

#256

Post by whoknew1122 »

One of a prior boss's war stories was that our CTO once logged in the main production server and did a 'rm -rf '. This is back when it'd actually delete everything. This was also back before most companies had back up prod servers.

From what I understand, it was a long weekend bringing that server up from tape backups.

The funniest thing to me wasn't that it happened. It was who it happened to. The CTO has created (and was granted patents for) multiple utilities still used very frequently in nix environments today.

Re: Ask HN: Best “I brought down production” story?

#257

Post by whoknew1122 »

gossamer wrote: post #148

Most of these stories seem to be from 20+ years ago. Do newer sysadmins just not make mistakes anymore? Or is 20 years the timespan that is needed to get over the embarassment? :-)

The mistakes made in these comments led to process improvements. Also, the price of computing has come down substantially such that you can actually build highly available and fault tolerant systems.

New sysadmins make mistakes, sure. But they're typically not as critical because recovery processes and correct system architecture insulate the admin from making a mistake with such catastrophic results.

Re: Ask HN: Best “I brought down production” story?

#258

Post by msiemens »

I brought down production for a project — by running the deployment CI pipeline for already deployed commit. A couple of minutes later, production was thoroughly dead.

Turned out that my coworker had set up the CI process to use a PHP-based zero downtime deployment scheme where each release was deployed into a folder with the commit hash as name and then a symlink was updated to point the web root to this new release folder.

But, critically, he also configured CI to delete old releases at the end of the deployment pipeline - by removing all release folders older than three days. And by re-deploying a commit older than three days, after uploading the code and updating the symlink, the release‘s folder was considered old and deleted at the end of the pipeline, leaving the webserver with an empty directory as web root.

Re: Ask HN: Best “I brought down production” story?

#259

Post by helmsb »

notatoad wrote: post #116
mansoon wrote: post #115

Earlier quoted context omitted.

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

It's not good to punish people for making mistakes in the course of their work (especially if that work is meant to be educational) It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access. Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the cons…

I’ve never understood the logic of firing someone over a mistake like that. They’re now the person least likely to make a similar mistake and they will maintain the institutional knowledge to help ensure it doesn’t happen again.

Re: Ask HN: Best “I brought down production” story?

#260

Post by helmsb »

I worked with someone who pushed out a config change to production and went home.

Soon, he got a call asking what he did because all of our credit card processing went down.

Shortly after that we got a call from our credit card processor (one of the largest in the country) asking what we did because it was causing a cascading failure in their systems and had taken down a big chunk of online credit cards processing including Apple and Walmarts websites.

We tracked it down to the commit which had accidentally nulled the a username for authenticating with our credit card processor. They had a previously unknown bug that caused a memory fault when the username was null. The processing was queue based so when one machine failed, another would pick it up and try and process it and would fail. This happened until the entire data center fell over.

A few of the lessons learned: 1. Always check the length of user inputs. 2. Build in circuit breakers to prevent cascading failures. 3. Don’t push code without review to production, minutes before leaving.

Everyone involved kept their jobs and we learned a lot of programming lessons that day.

Post reply on HN