Live data from Hacker News

Why HN was down

news.ycombinator.com

201–210 of 303 posts

Re: Why HN was down

#201
post #85

I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.

he's not legendary for his IT skills.

Now he is ;)

Re: Why HN was down

#202
post #104

Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exa…

use CHECK constraints to prevent invalid data patterns when possible.

Re: Why HN was down

#203

Earlier quoted context omitted.

* while you are displaying a tree keep track of the items you already displayed so you can detect a cycle

I think the assumption there was that it was safe, since the code disallowed this from happening, naturally.

Assumption is the mother of all screw ups.

Even if you think that the code that creates and modifies your data will not put it in some undesired state, the code that uses this data should assume that the data may be in all undesired states you can dream up and should do its best not to do something seriously bad when that happens (like landing in endless loop/recursion or executing possibly user provided strings).

Re: Why HN was down

#204
post #170

Earlier quoted context omitted.

A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces m…

Working from home I tend to just write out my thoughts on a piece of paper. It works perfectly.

When I work from home (and I did it exclusively for 8 months of last year) I ended up talking to my wife (my 1 yo daughter wouldn't stand still for long enough).

My wife is a social worker by training, so it was pretty rare (though not unheard of) for her to be able to give me real input, but over the years I've trained her well enough to follow most of what I'm saying and nod at the right points :)

Re: Why HN was down

#205

I use assertions to protect against things like this. I liberally sprinkle my code with assertions (CS theory calls them pre-conditions and post-conditions, iirc) to crash early if the system is an invalid state. One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.

What assertion would you have used in this case? For every comment you'd have to iterate through all it's parents to check if there is a cycle, which seems pretty inefficient to do for something that should never happen (there are other ways that you could check for this problem as you go, but the only other ways that I can think of require holding extra state just in order to perform the assertion). I'm for assertio…

typically, if you're operating upon a particular comment, you've gotten there by traversing to it from the parent. Ensuring that traversals don't encounter cycles is easy, keep hold of a hashtable (or a set) of comment ids as you traverse. As the traversal encounters a comment, its id is added to the hash, and as you complete traversal of each comment, the id is removed. If you encounter an id that's already in the set, assertion failed - or better yet, log the condition and then cease the traversal. That way everything keeps running and the error is visible in the logs.

If the code is organized (as it should be) such that all functions which require traversal of hierarchical comments pull this from a single function, then the hash check only need be applied in that one place in the code, where it need not be visible anywhere else.

Re: Why HN was down

#206
post #64

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time…

Yup, the incentive is there to state your problem as clearly as possible to get back a good response. By doing this I answer my own question half of the time.

Re: Why HN was down

#207
post #202
post #104

Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exa…

use CHECK constraints to prevent invalid data patterns when possible.

Sadly, even that isn't enough.

In our production database, I used CHECK constraints religiously. Worked great.

Then one day, I was no longer able to commit ANY transactions to a particular table, even completely innocuous ones.

The problem? The database itself had violated its own CHECK constraint on a previous commit, but was enforcing it on all subsequent commits, causing them to fail. Brilliant.

Moral: not even CHECK constraints will save you.

----

P.s. This was a proprietary database, and when I reported the problem to the vendor (eventually, I figured out how to reproduce it), the vendor actually refunded our (expensive) support contract rather than fix the bug -- they couldn't figure out how to fix it despite having a small bug report that reproduced the problem.

In the end, I actually had to remove the CHECK constraint altogether. :(

Re: Why HN was down

#209
post #174

Earlier quoted context omitted.

Curiously, that episode was on TV where I live just an hour ago.

http://en.wikipedia.org/wiki/Pareidolia

Oh, I know. Just wondered if the person who posted the comment above mine had just seen that particular episode, too.

Re: Why HN was down

#210
post #193
post #66

Earlier quoted context omitted.

I was thinking something like (supposing the comments were stored as "closure tables" like Karwin suggests): CREATE TABLE comment_tree ( ascestor_id REFERENCES comments(id) NOT NULL, descendant_id REFERENCES comments(id) NOT NULL, CHECK ( ascestor_id descendant_id ) ) but I'm probably overlooking something. (I'm aware that HN uses flat files, I was just making a counter-point to the "simple assert" solution...)

That will prevent a child being its own parent. It won't work for more than one level i.e a post being its own grandchild. Assume (post_id, parent_id) sequence: (1, 3) -> (2, 1) -> (3, 2).

you can assert that "post_id > parent_id", assuming comments are always created subsequent to the creation of their parents (as is the case here) and that integer identifiers are always increasing (otherwise use timestamps). (1, 3) above would indicate an invalid case (not necessarily a cycle, but a precondition for one).
Post reply on HN