Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#181

Post by cyphar »

linsomniac wrote: post #107
hamburglar wrote: post #62

Earlier quoted context omitted.

Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”

Probably not if it was overwritten (": >/bin/bash") rather than removed and recreated ("rm -f /bin/bash; : >/bin/bash"). The former will cause all processes to see the empty file, the latter would leave processes with access to the old contents. In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you…

On Linux you can't overwrite a binary that is currently being executed (you'll -ETXTBSY when you try to open it for writing, and truncate will also fail).

Re: Ask HN: Best “I brought down production” story?

#183

Post by syncdk »

brutal_chaos_ wrote: post #180
truetraveller wrote: post #106

Earlier quoted context omitted.

That's over engineering. At this point, just rely on PITR. FWIW, postgres does have a "reaper" via vacuum, but not for the purpose of safety. but rather to allow for mvcc.

Hi, if I may, what does PITR stand for?

https://en.wikipedia.org/wiki/Point-in-time_recovery

Re: Ask HN: Best “I brought down production” story?

#184

Post by cobertos »

I used to work in an automotive assembly plant doing IT. Me and a co-worker were out on the plant floor when he mentions a ping tool he found in the office he wanted to try out. He plugged it into a network drop and fired off a ping. The whole line stopped shortly afterwards but we hadn't connected the dots.

Lot of communication started happening on the radio, apparently all the stations were failing their quality checks. After a couple minutes, the plant manager, head of IT, and other people pulled up to where we were. Co-worker unplugged the device and plant slowly comes back up.

Turns out the ping tool was last used years ago to test connectivity to the new quality server and was configured with the static IP of said server. After the ping tool announced itself on the network, the real quality server stopped receiving the quality events so the line stopped.

Re: Ask HN: Best “I brought down production” story?

#185

Post by d1str0 »

ketzo wrote: post #149
peeters wrote: post #133

Earlier quoted context omitted.

> The person responsible for the systemic failure should see the consequences You don't see the contradiction of terms there? A systemic failure is by definition not the responsibility of one person. You're saying people should be able to make mistakes. But not those people.

A piece of the system (a junior developer) is allowed to make mistakes. The person responsible for architecting and protecting the system (the CTO)... less so.

That depends, this might have been this CTO’s first time as CTO. Without knowing the story, they could so have been pretty far out of their element and just lucky to be a founder or something.

Even C-level people always have to have their first day as C-level and of course they will make mistakes.

The important thing is learning from them of course.

Re: Ask HN: Best “I brought down production” story?

#186

Post by tsjq »

parkersweb wrote: post #31

This was 20 years ago now - it was my first day in a new job working for a startup. Our startup was based in the garden office of a large house and the production server was situated in a cupboard in the same room. The day I started was a cold January day and I’d had to cycle through flooded pathways to get to work that morning - so by the time I arrived my feet were soaked. Once I’d settled down to a desk I asked if…

quite close , the first few mins of this Apple WWDC from a few years ago . https://youtu.be/oaqHdULqet0?t=32

Re: Ask HN: Best “I brought down production” story?

#187

Post by samus »

maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

Someone hogging the database with an analytics query is a honest error because of an insidious footgun inherent in the technology stack. On the other hand, the CTO permitted access to the production database ... why? To learn MySQL, it would have been sufficient to set up a local instance. Or connect to testing/staging environments to get at some data.

Re: Ask HN: Best “I brought down production” story?

#188

Post by BozeWolf »

angrais wrote: post #57
BozeWolf wrote: post #49

About 15 years ago, when ssh-ing into servers was quite normal. In eterm on my gentoo linux laptop with enlightenment desktop I typed: su - shutdown -h now Because I was tired and I wanted to go to bed. Came back after brushing my teeth. F### laptops and linux! Screen still on. The thing didn’t shutdown! Strange thing was: in the terminal something said it got a shutdown signal. Then I realized I shutdown a remote se…

Is SSHing no longer normal? What do the cool cats do these days to manage their servers? I use K8s and docker to run software on my server, but initiate these via SSH. I suppose CI is perhaps modern approach or what else is everyone using?

With sshing i meant: ssh to server, install software and configure server. No docker, no ansible, nothing of that.

Re: Ask HN: Best “I brought down production” story?

#190

Post by philbo »

Wrote a stored procedure that joined with our `sessions` table. Worked fine in local dev and staging of course. And it was fine in prod too for 6 weeks or so because the other end of the join didn't have any rows yet.

Eventually some changes landed so the other table starts receiving data and the cost of the join begins to increase. Some time around midnight on a Friday night, performance degrades sufficiently for queries to start timing out. No-one can read a session, the site is effectively down for everyone (30m DAU). Ops team lost their whole weekend tracing back to what I'd broken, because the delay since deployment confused the issue so much.

Post reply on HN