Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 303 posts

Re: Why HN was down

#121

Post by dap »

irahul wrote: post #72
lucb1e wrote: post #37

Earlier quoted context omitted.

Upvoted. This is what I wanted to reply, but then thought better of it and moderated my response.

I think you people already know the answer. The amount of freedom and stake/reward system for pg is different from yours. I can't speak for pg, but personally, I am not going to write a migration to re-parent a single thread if the site in question is my side project, doesn't bring revenue, has some intangible benefits, but not so much that warrant putting much labor into it. Either it would be `thread.parent = new_p…

The problem is thinking of it as the cost of implementing the feature vs. doing manual surgery on the production database, without realizing that if you choose the latter, you're also choosing the risk that you'll spend hours debugging the system when the surgery goes wrong. It's a tradeoff, to be sure, but it's not clear that the latter is cheaper on expectation.

Re: Why HN was down

#122

Post by wpietri »

jgrahamc wrote: post #102
neurotech1 wrote: post #93

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

"No deflecting blame" Who were they going to blame?

He could have blamed the new server. Or whatever distracted him. Or the user, for being dumb. I've seen people do all of those.

Or he could have just dodged the blame entirely.

Re: Why HN was down

#123

Post by lmm »

xentronium wrote: post #26

Whoa, what an unfortunate coincidence. This whole bug would be so much easier to find, if it weren't for the new server.

The bugs that actually hit production are always like this - a confluence of three or so factors - because if it were simpler you'd have caught it earlier.

(Though I have to say, upgrading the code at the same time as you're restarting to fix a problem is really a rookie mistake. It's incredibly tempting because it saves so much time, but if you do it you will get it wrong sooner or later. One of the hardest skills in programming is acquiring that zen that you need to wait in a state of readiness for the effects of your first change to make themselves apparent, rather than changing something else)

Re: Why HN was down

#124

Post by harrisreynolds »

Just about anyone that has programmed for any length of time has done something like this. It is one of those "fixes" that after it's actually fixed you try to never think of it again. Good to know PG is mortal. :-)

Re: Why HN was down

#125

Post by smackfu »

neurotech1 wrote: post #93

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

I don't know, it's a lot easier to be transparent when the stakes are so low. Most service providers have a real incentive to not put out quotes that can later be used against them, which tends to make explanations very technical or deflecting.

Re: Why HN was down

#126

Post by [deleted] »

z-e-r-o wrote: post #64
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time…

[deleted]

Re: Why HN was down

#127

Post by JohnBooty »

hnriot wrote: post #71
JohnBooty wrote: post #53

Earlier quoted context omitted.

I disagree. Very few companies would think negatively of an engineer if they made such a mistake on a non-essential, non-revenue-generating fun/research project. How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!) I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from those guys! They had discussion forum downtime."

They could save even more if they shut it down! That's a ridiculous thing to say. We could all save money that way. You've obviously never worked in a for profit corporation. in such there are policies and practices put in place to prevent just this kind of newbie mistake. You never modify the live database directly. Never ever. Whether it's a bottom line property or not. I didn't say it would negatively impact YC's…

"You've obviously never worked in a for profit corporation"

Wow, really? I don't think that attitude is warranted at all.

At any company (for-profit or otherwise) there is a finite amount of time and money -- and surely we can agree that solid development/deployment practices carry an upfront time/money cost, can't we?

In an ideal world, all projects would have continuous build processes, automated tests, and management tools extensive enough to render live database surgery unnecessary.

Perhaps you've worked at companies so flush with cash that every single line of code, research project or otherwise, has gone through rigorous development/testing/deployment practices. If so, I'm jealous. I've always worked at companies that had to be choosey about how they spend their resources.

Re: Why HN was down

#128

Post by reddit_clone »

jgrahamc wrote: post #102
neurotech1 wrote: post #93

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

"No deflecting blame" Who were they going to blame?

The user who replied incorrectly?

Re: Why HN was down

#129

Post by elcodedocle »

z-e-r-o wrote: post #64
DanielBMarkham wrote: post #7

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging

For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time…

Same here: I've been working on a couple of projects by myself for the most part of last year, and when even the duck failed, I could usually figure out an answer just by trying to find the words to post my problem in SO in a way somebody would take the time to read it and be able to answer it. I don't recommend it as a first approach, though, since it's quite time consuming (Or maybe I should blame it on not being a native speaker...)

Re: Why HN was down

#130

Post by Uchikoma »

Appreciating the details.

"Hacker News was down all last night."

With the internet there is no "last night" ;-) Europe - and more so Asia I assume - had to live for many working hours without HN.

Post reply on HN