Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 303 posts

Re: Why HN was down

#221

Post by artursapek »

dguaraglia wrote: post #212
wpietri wrote: post #122

Earlier quoted context omitted.

He could have blamed the new server. Or whatever distracted him. Or the user, for being dumb. I've seen people do all of those. Or he could have just dodged the blame entirely.

Some people are just freaking difficult to work with. I've worked with people that wouldn't accept responsibility even after every other possible cause was ruled out. I've even gotten this reply: "well, you must have been unlucky to get the faulty e-mail, because it seems to work most of the time". Yeah, because that's how programming works: cowboy coding and hoping for the best. This guy actually called bugfixes "op…

pg has an essay where he says the smartest people he knows are always willing to take blame or admit they don't know the answer to a question.

Re: Why HN was down

#222

Post by Someone »

irahul wrote: post #179
Someone wrote: post #79

Earlier quoted context omitted.

A constraint on the parent-child link table "Child creation time stamp > Parent creation timestamp" would do it. Might not be a bad idea, if the site were to have the two requirements "maintenance must be done on the live site from a repl" and "5 nines availability".

How are you modelling your data? I think this should be a self reference. create table post (id int primary_key, parent_id int references post(id), child_id int references post(id), created_at timestamp) How will you place the check constraint? You only have parent_id and child_id, not parent and child entities. You will have to write a trigger. I am not saying this can't or shouldn't be done. I am saying a db won't…

Following http://stackoverflow.com/questions/3438066/check-constraint-..., and assuming that ID's get doled out in increasing order:

    create table post (
      id int primary_key,
      parent_id int references post(id),
      CONSTRAINT foo CHECK (id > parent_id)
    );

Re: Why HN was down

#223

Post by oh_sigh »

badgar wrote: post #131
oh_sigh wrote: post #94

Earlier quoted context omitted.

> pg just runs this site out of the good of his heart Hilarious. I would have believed you if you appended "and his wallet"

One core. One HD. Bandwidth is trivial with no images. How much do you think this site costs to run?

You didn't understand what I said. I was implying that this site is a money-maker for pg, not that it costs him money.

Re: Why HN was down

#224

Post by lifeisstillgood »

philsnow wrote: post #196
lifeisstillgood wrote: post #67

There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…

s/anime/animus/ ? or is this a new usage of "anime"?

s/anime/anima - as in soul, vitality

(Not so much Jung's inner woman)

I think the sentence does read better if it is complaining there are not enough cyberpunk Japanese comics on my site though :-)

Re: Why HN was down

#225

Post by paganel »

dguaraglia wrote: post #212
wpietri wrote: post #122

Earlier quoted context omitted.

He could have blamed the new server. Or whatever distracted him. Or the user, for being dumb. I've seen people do all of those. Or he could have just dodged the blame entirely.

Some people are just freaking difficult to work with. I've worked with people that wouldn't accept responsibility even after every other possible cause was ruled out. I've even gotten this reply: "well, you must have been unlucky to get the faulty e-mail, because it seems to work most of the time". Yeah, because that's how programming works: cowboy coding and hoping for the best. This guy actually called bugfixes "op…

> And the first time they avoid taking responsibility for something that was clearly their fault, fire them.

I guess there's still difficult for a lot of people to acknowledge their own mistakes, maybe because they're afraid of getting fired for that (acknowledging the mistake), which in the case of startups/small companies happens very rarely.

From my own experience of working at startups for my entire professional career as a programmer (7 and a half years) I can tell you that the first step when noticing you f.cked something up is to take immediate responsibility and then asking yourself "how can I/we fix this?" (you might need the help of other people to fix your mistake). After you've fixed the issue the question should be "how can we make so that this doesn't happen again?". That being solved I'd say nobody cares anymore whose fault was it to begin with, there's always other more important stuff to do.

I agree that maybe at larger companies this kind of thing might happen exactly the opposite way, i.e. you can get fired for making a mistake and nobody really cares to fix other people's stuff, because their next paycheck/financial well-being does not depend on that (or so they think).

Re: Why HN was down

#226

Post by vertis »

jgrahamc wrote: post #102
neurotech1 wrote: post #93

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

"No deflecting blame" Who were they going to blame?

Sequoia Capital or Andreessen Horowitz

Re: Why HN was down

#227

Post by jonob7 »

mafro wrote: post #198
3pt14159 wrote: post #183

Earlier quoted context omitted.

I've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have select count(distinct column_name) having count(distinct column_name) > 1; To make sure that there isn't an underlying uniqueness assumption. Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one…

This. Back in the day when I was in more of an analyst role, I ended up /having/ to hack on the live DB frequently (reasons for this were myriad). 1. Always, always make a backup just before the hack. 2. Write a small set queries like 3pt14159's to check uniqueness and other pertinent properties. 3. Write a SELECT query to show the data you are going to change. 4. Borrow the WHERE clause from 3, and write your UPDATE…

This is pretty much exactly how I do it.

I still sometimes get that sinking feeling in the stomach that I have screwed something up, usually just after I hit the 'execute' button. And I really don't want to have to take the site down to run the restoration.

Re: Why HN was down

#228

Post by lalc »

lifeisstillgood wrote: post #224
philsnow wrote: post #196

Earlier quoted context omitted.

s/anime/animus/ ? or is this a new usage of "anime"?

s/anime/anima - as in soul, vitality (Not so much Jung's inner woman) I think the sentence does read better if it is complaining there are not enough cyberpunk Japanese comics on my site though :-)

Here I was thinking you were referring to the Japanese meme "No ___, No Life!"

Re: Why HN was down

#230

Post by swah »

luser001 wrote: post #27
swah wrote: post #18

Earlier quoted context omitted.

The kind of assertion he needed though, could only be ensured by the database, not application code (my impression).

Agreed, infinite loops are a little hard to protect using asserts. When I hit the first infinite loop bug on a code path, I frequently add code to assert that the number of calls is less than $A_LARGE_NUMBER to catch future occurrences of the same root cause.

This is similar to the "while with timeout" that is common in embedded code (of course, watchdogs are better...)
Post reply on HN