I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.
he's not legendary for his IT skills.
Why HN was down
201–210 of 303 posts
Re: Why HN was down
#202Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exa…
Re: Why HN was down
#203Earlier quoted context omitted.
* while you are displaying a tree keep track of the items you already displayed so you can detect a cycle
I think the assumption there was that it was safe, since the code disallowed this from happening, naturally.
Even if you think that the code that creates and modifies your data will not put it in some undesired state, the code that uses this data should assume that the data may be in all undesired states you can dream up and should do its best not to do something seriously bad when that happens (like landing in endless loop/recursion or executing possibly user provided strings).
Re: Why HN was down
#204Earlier quoted context omitted.
A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces m…
Working from home I tend to just write out my thoughts on a piece of paper. It works perfectly.
My wife is a social worker by training, so it was pretty rare (though not unheard of) for her to be able to give me real input, but over the years I've trained her well enough to follow most of what I'm saying and nod at the right points :)
Re: Why HN was down
#205I use assertions to protect against things like this. I liberally sprinkle my code with assertions (CS theory calls them pre-conditions and post-conditions, iirc) to crash early if the system is an invalid state. One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.
What assertion would you have used in this case? For every comment you'd have to iterate through all it's parents to check if there is a cycle, which seems pretty inefficient to do for something that should never happen (there are other ways that you could check for this problem as you go, but the only other ways that I can think of require holding extra state just in order to perform the assertion). I'm for assertio…
If the code is organized (as it should be) such that all functions which require traversal of hierarchical comments pull this from a single function, then the hash check only need be applied in that one place in the code, where it need not be visible anywhere else.
Re: Why HN was down
#206Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging
For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time…
Re: Why HN was down
#207Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exa…
use CHECK constraints to prevent invalid data patterns when possible.
In our production database, I used CHECK constraints religiously. Worked great.
Then one day, I was no longer able to commit ANY transactions to a particular table, even completely innocuous ones.
The problem? The database itself had violated its own CHECK constraint on a previous commit, but was enforcing it on all subsequent commits, causing them to fail. Brilliant.
Moral: not even CHECK constraints will save you.
----
P.s. This was a proprietary database, and when I reported the problem to the vendor (eventually, I figured out how to reproduce it), the vendor actually refunded our (expensive) support contract rather than fix the bug -- they couldn't figure out how to fix it despite having a small bug report that reproduced the problem.
In the end, I actually had to remove the CHECK constraint altogether. :(
Re: Why HN was down
#208Re: Why HN was down
#209Re: Why HN was down
#210Earlier quoted context omitted.
I was thinking something like (supposing the comments were stored as "closure tables" like Karwin suggests): CREATE TABLE comment_tree ( ascestor_id REFERENCES comments(id) NOT NULL, descendant_id REFERENCES comments(id) NOT NULL, CHECK ( ascestor_id descendant_id ) ) but I'm probably overlooking something. (I'm aware that HN uses flat files, I was just making a counter-point to the "simple assert" solution...)
That will prevent a child being its own parent. It won't work for more than one level i.e a post being its own grandchild. Assume (post_id, parent_id) sequence: (1, 3) -> (2, 1) -> (3, 2).