Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 303 posts

Re: Why HN was down

#91

Post by stcredzero »

birken wrote: post #60

Do you have munin monitoring on the production HN server? That would really make situations like this easier to debug. First, it can pinpoint exactly when something started happening, which in this situation might have helped you realize the problem was caused by your change. Secondly, in this specific situation it probably would have been easier to differentiate a situation where you are running low on memory vs thi…

That and something like the Smalltalk Change Log would have made this a no-brainer debug. (Yes, every REPL action in Smalltalk got logged by the same mechanism that logged every code change.) Such mechanisms aren't trivial, but they're not rocket science either, and they have tremendous ROI.

Re: Why HN was down

#92

Post by jlgreco »

badgar wrote: post #32
benatkin wrote: post #3

So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?

> So what do you do to avoid this in the future? It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.

> there's no postmortems

1. Press [Home] key.

2. Read postmortem.

3. ???

Re: Why HN was down

#93

Post by neurotech1 »

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes.

Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

Re: Why HN was down

#94

Post by oh_sigh »

badgar wrote: post #32
benatkin wrote: post #3

So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?

> So what do you do to avoid this in the future? It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.

> pg just runs this site out of the good of his heart

Hilarious. I would have believed you if you appended "and his wallet"

Re: Why HN was down

#95

Post by chernevik »

irahul wrote: post #81
lucb1e wrote: post #70

Earlier quoted context omitted.

I sort of meant that you shouldn't modify things like that directly. Be it a filesystem, database, or any other place that makes it possible to mess things up to bring a rather strong server down.

I see where you are coming from. But I am saying this didn't happen because he did things live. This happened because he entered incorrect id making a thread its own parent(or grandparent; doesn't matter). This is the kind of mistake one would make even if you were writing proper migrations. He was doing things live isn't an issue; neither is an incorrect id. The issue is the code doesn't check for loops.

I am but an egg, I have two questions.

One, if the data were held in a database, should a change like this be captured in the database logs? I am seeing more and more situations where I want these, I notice that they are by default turned off for mysql and wonder if this reflects a de facto judgment that logging slows performance more than is usually worthwhile.

Two, if the data were kept in a database, wouldn't something like this be prevented by a constraint preventing a comment from making itself an ancestor? But I suppose there is a slight performance hit in checking such constraints, and the case arises so rarely that this hit isn't generally worthwhile.

Re: Why HN was down

#96

Post by hnriot »

irahul wrote: post #72
lucb1e wrote: post #37

Earlier quoted context omitted.

Upvoted. This is what I wanted to reply, but then thought better of it and moderated my response.

I think you people already know the answer. The amount of freedom and stake/reward system for pg is different from yours. I can't speak for pg, but personally, I am not going to write a migration to re-parent a single thread if the site in question is my side project, doesn't bring revenue, has some intangible benefits, but not so much that warrant putting much labor into it. Either it would be `thread.parent = new_p…

no, you just leave it alone, it would have sorted itself out if nothing had been done at all.

failing that you put a cap in the code that generates the page. Simple stuff.

Re: Why HN was down

#97

Post by banachtarski »

furyofantares wrote: post #46
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

There is a line from Futurama that perfectly applies ton a lot of debugging. Farnsworth: My God, is it really possible? Fry: It must be possible, it's happening. Fry: By the way, what's happening?

Is this the forward time machine episode??

I love futurama more than any man could love any tv show.

Re: Why HN was down

#98

Post by RyanMcGreal »

chernevik wrote: post #82
RyanMcGreal wrote: post #63

Earlier quoted context omitted.

I'm not sure if rubber duck debugging would have helped here. The problem was in the data, not the code. (I know, I know: in Lisp code is data.)

That's exactly the sort of thing a duck will tell you. "So what has changed? Let's see, new code, new server, and I fixed the commenter's comment. That was simple I just hard-hacked the comment id and . . . excuse me a second."

Good point. I was thinking in terms of going through the code line by line, which if anything would lead you away from the trail.

Re: Why HN was down

#99

Post by danielweber »

stcredzero wrote: post #68
larrys wrote: post #39

Earlier quoted context omitted.

Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.

There must be a rare personality type that never experiences this kind of overconfidence. Perhaps a less glamorous cousin to the Buddhist beginner's mind?

There are certain classes of autistic people who are very good at always following the rules and finding people who are not. And in certain places they are exactly what you want.

http://www.nytimes.com/2012/12/02/magazine/the-autism-advant...

Post reply on HN