Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 303 posts

Re: Why HN was down

#71

Post by hnriot »

JohnBooty wrote: post #53
hnriot wrote: post #14

it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.

I disagree. Very few companies would think negatively of an engineer if they made such a mistake on a non-essential, non-revenue-generating fun/research project. How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!) I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from those guys! They had discussion forum downtime."

They could save even more if they shut it down! That's a ridiculous thing to say. We could all save money that way.

You've obviously never worked in a for profit corporation. in such there are policies and practices put in place to prevent just this kind of newbie mistake. You never modify the live database directly. Never ever. Whether it's a bottom line property or not.

I didn't say it would negatively impact YC's business. It might make them look incompetent, but these things happen, but people don't approach YC for their website savvy, they go there for the money and the connections. Most of the VC firms i've EIR'd at have much worse IT than hn. Their sites are barely usable. It seems to just go with the territory.

Let's not be so defensive, PG can do what he likes with his site, including take it down whenever he feels like saving bandwidth. But in the real world these kinds of things get real people on a fast track to their exit interview.

Re: Why HN was down

#72

Post by irahul »

lucb1e wrote: post #37
hnriot wrote: post #14

it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.

Upvoted. This is what I wanted to reply, but then thought better of it and moderated my response.

I think you people already know the answer. The amount of freedom and stake/reward system for pg is different from yours.

I can't speak for pg, but personally, I am not going to write a migration to re-parent a single thread if the site in question is my side project, doesn't bring revenue, has some intangible benefits, but not so much that warrant putting much labor into it.

Either it would be `thread.parent = new_paret_id`; or if it occurred to me that it might introduce a loop, changing `parent=` to take loops into account followed by `thread.parent = new_parent_id`. What were you expecting? A bug tracker discussion, code commit, review, change request and deployment?

Re: Why HN was down

#73

Post by biot »

DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

People at work are amazed when I successfully debug an issue over the phone. In reality, it amounts to 50% experience plus another 50% of Sherlock Holmes: "When you have eliminated the impossible, whatever remains, however improbable, must be the truth". Once you've identified what you're dealing with via a few strategic questions, it becomes simple quite rapidly.

Re: Why HN was down

#74

Post by aroman »

z-e-r-o wrote: post #64
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time…

Likewise for me, but with IRC. Though I suppose I should try asking on SO first to save myself the semi-public embarrassment ;)

Re: Why HN was down

#75

Post by martinced »

lucb1e wrote: post #8

Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

"Are you saying you manually modify the database?"

Oh manually modifying production database on the fly ain't unheard of.

However it's still not "very Chuck Norris" on a scale of Chuck Norrisness compared to the modification of a running app directly in the REPL. I mean: it doesn't matter if you manually modify the DB itself or not when you directly modify the app from the REPL itself (the app being anyway "in charge" of the DB).

Sure, modifying manually the production DB might be an issue to some. But I can guarantee you that it's the last of your worries when you're actually modifying production code directly from the REPL ; )

Re: Why HN was down

#77

Post by triplepoint217 »

lucb1e wrote: post #70
irahul wrote: post #30

Earlier quoted context omitted.

HN runs on plain files. He wasn't modifying database, but calling functions(I believe) in the repl to change the parent id of the thread. But that apart, even if there were an admin button to change the parent id of a thread, he would still have made the same mistake. Unless the code in question was checking for loops. In that case, repl would have worked the same.

I sort of meant that you shouldn't modify things like that directly. Be it a filesystem, database, or any other place that makes it possible to mess things up to bring a rather strong server down.

Yes, I was trying to make posts editable on the HN instance I run, so I got clever and started messing with the files in emacs. Then I learned that the HN code does not like files in the story directory with ~ on the end of their name (emacs backup files), oops ;).

Re: Why HN was down

#78

Post by stcredzero »

lifeisstillgood wrote: post #67

There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…

> It read like a High School essay - no life, no anime. ... I am not entirely sure those thoughts are joined up (I am procrasting like crazy)

Your procrasting like crazy has much anime.

Re: Why HN was down

#79

Post by Someone »

irahul wrote: post #35
swah wrote: post #18

Earlier quoted context omitted.

The kind of assertion he needed though, could only be ensured by the database, not application code (my impression).

The kind of assertion he needed could not be ensured by the database. The kind of assertion he needed was there are no cycles in the graph. How would you ensure that in a database? Also, HN uses flat files, not database.

A constraint on the parent-child link table "Child creation time stamp > Parent creation timestamp" would do it.

Might not be a bad idea, if the site were to have the two requirements "maintenance must be done on the live site from a repl" and "5 nines availability".

Re: Why HN was down

#80

Post by DanielBMarkham »

RyanMcGreal wrote: post #63
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

I'm not sure if rubber duck debugging would have helped here. The problem was in the data, not the code. (I know, I know: in Lisp code is data.)

Yep. I thought this through as I was typing my comment.

(There must be some joke involving the use of a meta-duck, but I can't come up with it. :) (Same principle applies, of course, just LISP makes the determining of "what" a bit more tricky. (insert discussion here about the general differences between debugging imperative and functional code)))

Post reply on HN