That explains something weird I saw.
If I went to Google's cached copy, I could see threads, and then click on them. But the front page was down. But I could see individual threads.
Very confusing.
Why HN was down
41–50 of 303 posts
Re: Why HN was down
#42Related question: what is the timing for the 'Reply' link to show up? I might be fantasizing but sometimes it takes 5, sometimes 10 minutes to appear, leading people to reply as a sibling instead.
Deeply nested comments tend to be hot, and so the reply link takes a while to show up to try to give people some time to think about what they're going to say.
Re: Why HN was down
#43I guess it's time for NewRelic to add an Arc agent.
Re: Why HN was down
#44Earlier quoted context omitted.
The kind of assertion he needed though, could only be ensured by the database, not application code (my impression).
Agreed, infinite loops are a little hard to protect using asserts. When I hit the first infinite loop bug on a code path, I frequently add code to assert that the number of calls is less than $A_LARGE_NUMBER to catch future occurrences of the same root cause.
> Agreed, infinite loops are a little hard to protect using asserts.
assert(is_tree(comment_graph))
The database layer often contains this logic, but it depends on how you're building your application; NoSQL backends for example typically must put validation in the application layer. Since HN just uses files, a well-developed application layer should be riddled with invariants like this.
Re: Why HN was down
#45The nice thing about surgery with a computer program on a server is that death is not permanent.
Re: Why HN was down
#46Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging
There is a line from Futurama that perfectly applies ton a lot of debugging.
Farnsworth: My God, is it really possible?
Fry: It must be possible, it's happening.
Fry: By the way, what's happening?
Re: Why HN was down
#47PG, quick question: Did this impact the server hosting the YC Summer 2013 applications?
When I tried to edit mine, it simply said "Thanks, scotthtaylor"
Re: Why HN was down
#48So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?
> So what do you do to avoid this in the future? It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
"pg just runs this site out of the good of his heart"
Don't think it's a "good of his heart situation". HN provides a benefit to YC and YC companies and attracts people to YC. As another example Fred Wilson has a very popular blog AVC and has said many times that he considers it "his secret weapon" (or something like that) because the value it provides over his competition.
Re: Why HN was down
#49It makes me feel good knowing better programmers than me go through the same issues I face. :)
Re: Why HN was down
#50Earlier quoted context omitted.
I think I get what you're hinting at. Ok, so this is Hacker News , it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.
No that's right, but I worry about apache2 being down for potentially one or two users or bots that visit/crawl my website during a one-minute reboot. Meanwhile the big boys are down for 16 hours because they do things that any other person would have gotten a decent scolding for. Just look at the points per hour this thread is getting, if I had posted this about my website on my website people would have said I was…
I'd say this is a pretty important lesson: you don't need flawless technology and zero downtime to be popular and/or profitable. You need content worth viewing. People are more than willing to put up with technical errors if it's something they want/need.
Focus on providing people what they want/need, and don't worry so much about having flawless technology until you can employ a horde of PHDs.