Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 303 posts

Re: Why HN was down

#81

Post by irahul »

lucb1e wrote: post #70
irahul wrote: post #30

Earlier quoted context omitted.

HN runs on plain files. He wasn't modifying database, but calling functions(I believe) in the repl to change the parent id of the thread. But that apart, even if there were an admin button to change the parent id of a thread, he would still have made the same mistake. Unless the code in question was checking for loops. In that case, repl would have worked the same.

I sort of meant that you shouldn't modify things like that directly. Be it a filesystem, database, or any other place that makes it possible to mess things up to bring a rather strong server down.

I see where you are coming from. But I am saying this didn't happen because he did things live. This happened because he entered incorrect id making a thread its own parent(or grandparent; doesn't matter).

This is the kind of mistake one would make even if you were writing proper migrations. He was doing things live isn't an issue; neither is an incorrect id. The issue is the code doesn't check for loops.

Re: Why HN was down

#82

Post by chernevik »

RyanMcGreal wrote: post #63
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

I'm not sure if rubber duck debugging would have helped here. The problem was in the data, not the code. (I know, I know: in Lisp code is data.)

That's exactly the sort of thing a duck will tell you. "So what has changed? Let's see, new code, new server, and I fixed the commenter's comment. That was simple I just hard-hacked the comment id and . . . excuse me a second."

Re: Why HN was down

#83

Post by larrys »

stcredzero wrote: post #68
larrys wrote: post #39

Earlier quoted context omitted.

Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.

There must be a rare personality type that never experiences this kind of overconfidence. Perhaps a less glamorous cousin to the Buddhist beginner's mind?

I find that if I am doing something "dangerous" (example might be using power tools) I have to say to my self "be careful this is dangerous" to avoid being on autopilot and making casual errors. Maybe a better example is the way you train yourself after you've picked up a box the wrong way and pull something to try to remember each and every time to watch your specific movements.

Re: Why HN was down

#84

Post by dasil003 »

larrys wrote: post #39
dasil003 wrote: post #13

I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.

Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.

Too bad we rarely get a postmortem on batshit insane production hackery that actually goes off without a hitch.

Re: Why HN was down

#85

Post by hnriot »

dasil003 wrote: post #13

I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.

he's not legendary for his IT skills.

Re: Why HN was down

#86

Post by Semaphor »

z-e-r-o wrote: post #64
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time…

That happens to me a lot. Most of the time I just formulate the question I have in my head into something coherent and by that point I either have the solution, know what to search for or, in case it's not a question but a comment, I realize it's not worth saying.

Re: Why HN was down

#87

Post by dasil003 »

larrys wrote: post #83
stcredzero wrote: post #68

Earlier quoted context omitted.

There must be a rare personality type that never experiences this kind of overconfidence. Perhaps a less glamorous cousin to the Buddhist beginner's mind?

I find that if I am doing something "dangerous" (example might be using power tools) I have to say to my self "be careful this is dangerous" to avoid being on autopilot and making casual errors. Maybe a better example is the way you train yourself after you've picked up a box the wrong way and pull something to try to remember each and every time to watch your specific movements.

I have to remind myself this every time I start up Sequel Pro now since in the last release they switched the command keys for Run Selected... and Run All...

Re: Why HN was down

#88

Post by cube13 »

RyanMcGreal wrote: post #63
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

I'm not sure if rubber duck debugging would have helped here. The problem was in the data, not the code. (I know, I know: in Lisp code is data.)

Rubber duck debugging may have actually been the distraction that caused pg to make the mistake, too!

Re: Why HN was down

#89

Post by luser001 »

lifeisstillgood wrote: post #67

There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…

> I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing.

IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc.

Anyway, that's why I like sudo.

Post reply on HN