Why HN was down • Hacker News Board

Re: Why HN was down

#21

Post by irahul » Mon, Feb 18, 2013, 6:43 PM UTC

Disclaimer: Hindsight is 20/20, and stuff.

If reverting code didn't fix it, reverting server didn't fix it, incorrect data is the most likely culprit(I am not claiming this should have outright occurred to you; just thinking out loud). I take it you introduced non terminating recursion by making a thread its own parent, and you made the change on disk.

But this analysis is the last thing that comes to mind when you already have introduced 2 new variables the same day - new code, new server. And an old, recurring variable(GCing too much) is in play as well.

Re: Why HN was down

#22

Post by sgt » Mon, Feb 18, 2013, 6:43 PM UTC

lucb1e wrote: post #8

Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

I think I get what you're hinting at.

Ok, so this is Hacker News, it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.

Re: Why HN was down

#23

Post by hyh1048576 » Mon, Feb 18, 2013, 6:44 PM UTC

The "More" button at the bottom is still not clickable sometimes. Get a "Unknown or expired link." message.

Re: Why HN was down

#24

Post by andreasklinger » Mon, Feb 18, 2013, 6:44 PM UTC

I appreciate (if not love) the fact that you bugfix and server-change yourself.

True hacker spirit.

Re: Why HN was down

#25

Post by hyh1048576 » Mon, Feb 18, 2013, 6:44 PM UTC

The "More" button at the bottom is still not clickable sometimes. Get a "Unknown or expired link." message.

Re: Why HN was down

#26

Post by xentronium » Mon, Feb 18, 2013, 6:44 PM UTC

Whoa, what an unfortunate coincidence. This whole bug would be so much easier to find, if it weren't for the new server.

Re: Why HN was down

#27

Post by luser001 » Mon, Feb 18, 2013, 6:45 PM UTC

swah wrote: post #18

luser001 wrote: post #12

I use assertions to protect against things like this. I liberally sprinkle my code with assertions (CS theory calls them pre-conditions and post-conditions, iirc) to crash early if the system is an invalid state. One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.

The kind of assertion he needed though, could only be ensured by the database, not application code (my impression).

Agreed, infinite loops are a little hard to protect using asserts.

When I hit the first infinite loop bug on a code path, I frequently add code to assert that the number of calls is less than $A_LARGE_NUMBER to catch future occurrences of the same root cause.

Re: Why HN was down

#28

Post by naturalethic » Mon, Feb 18, 2013, 6:47 PM UTC

If the problem existed before the code update, why would you assume it was the code update that caused the problem?

Re: Why HN was down

#29

Post by lucb1e » Mon, Feb 18, 2013, 6:47 PM UTC

sgt wrote: post #22

lucb1e wrote: post #8

Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

I think I get what you're hinting at. Ok, so this is Hacker News , it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.

No that's right, but I worry about apache2 being down for potentially one or two users or bots that visit/crawl my website during a one-minute reboot. Meanwhile the big boys are down for 16 hours because they do things that any other person would have gotten a decent scolding for. Just look at the points per hour this thread is getting, if I had posted this about my website on my website people would have said I was stupid.

You are right though, making mistakes is human as they say, and nobody dies because of this. In fact, less popularity might be good for the site's content quality. I'm just surprised by how much they care about thousands of hourly users, that what I would dream of having.

Re: Why HN was down

#30

Post by irahul » Mon, Feb 18, 2013, 6:49 PM UTC

lucb1e wrote: post #8

Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

HN runs on plain files. He wasn't modifying database, but calling functions(I believe) in the repl to change the parent id of the thread.

But that apart, even if there were an admin button to change the parent id of a thread, he would still have made the same mistake.

Unless the code in question was checking for loops. In that case, repl would have worked the same.