Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 303 posts

Re: Why HN was down

#101

Post by chc »

lucb1e wrote: post #29
sgt wrote: post #22

Earlier quoted context omitted.

I think I get what you're hinting at. Ok, so this is Hacker News , it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.

No that's right, but I worry about apache2 being down for potentially one or two users or bots that visit/crawl my website during a one-minute reboot. Meanwhile the big boys are down for 16 hours because they do things that any other person would have gotten a decent scolding for. Just look at the points per hour this thread is getting, if I had posted this about my website on my website people would have said I was…

Patrick McKenzie had a great horror story on his blog a couple of years back. He runs a service that provides appointment reminders to businesses' clients (e.g. "Don't forget, you have an appointment to get your hair colored at Best Little Hair House tomorrow at 3"). Long story short, an attempt to manually correct a hangup in the live system resulted in his product spamming his customers' clients (that's right — not just his customers, but their customers) with up to 40 phone calls back-to-back.

So, how many customers do you think he lost because of this? The answer is two, and one of them signed back up because they were impressed by the great job he did in handling the fiasco.

Moral of the story: As long as you really are making your best effort, you might be surprised how willing people are to deal with human error. Yes, they might be be mad, but a mistake is (usually) not the end of the world.

Re: Why HN was down

#102

Post by jgrahamc »

neurotech1 wrote: post #93

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

"No deflecting blame"

Who were they going to blame?

Re: Why HN was down

#103

Post by wpietri »

hnriot wrote: post #14

it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.

Dumb companies, maybe.

The goal was reasonable. The action was reasonable. What are you firing somebody for? Making mistakes? Good luck making that a hiring criterion. "Ok, tell us about a time you made a mistake and what you learned from it. What's that? You never have made one? Great, you're hired!"

The solution from a retrospective should never be, "Let's make people more scared to do the right thing." Or "Let's fire people with bad luck." Firing people of PG's caliber isn't a solution, it's just another problem.

Re: Why HN was down

#104

Post by sehugg »

Great postmortem and good lessons to learn here:

* Don't manually modify database without a well-tested procedure and another pair of eyes

* Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms

* Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem)

I'm pretty sure I've repeated this exact same sequence before with similar results...

Re: Why HN was down

#106

Post by tolmasky »

Why do "self posts" like this show up in the same light gray as posts with negative vote counts? My eyes aren't great and I find it hard to read

Re: Why HN was down

#108

Post by ww520 »

benatkin wrote: post #3

So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?

This reminds me of the countless conversation I had with people after a crisis. What can we do to prevent this from happening again? What process can we put in place? What restriction needed to be tightened up?

And that's how processes are born.

Re: Why HN was down

#110

Post by UnoriginalGuy »

luser001 wrote: post #89
lifeisstillgood wrote: post #67

There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…

> I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc. Anyway, that's why I like sudo.

Plus security. With root login disabled a remote attacker won't have a known username to attack.
Post reply on HN