Are you sure that comment's name wasn't Phillip J Fry?
Apologies for the trivial comment
Why HN was down
291–300 of 303 posts
Re: Why HN was down
#292Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exa…
use CHECK constraints to prevent invalid data patterns when possible.
That's hard to do when you are afraid of databases and just store everything in files.
Re: Why HN was down
#293Earlier quoted context omitted.
he's not legendary for his IT skills.
It was his IT skills that got him the big sale to Yahoo that got him the bucks to start YC. Not sure where this comment came from.
You really think yahoo bought viaweb because of pg's legendary ability to reboot servers and type "./configure && make && sudo make install"?
Re: Why HN was down
#294Earlier quoted context omitted.
A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces m…
I call it the House method :) You bring a detailed problem and break it down, and talk about it to someone else (who often isn't qualified to answer your questions due to knowledge/time constraints) - and in doing so - resolve the problem by challenging one's own assumptions. This was effectively how every House episode was resolved.
Haha, we called it the House method too, we ended up making a cardboard humanoid for when nobody was available.
Re: Why HN was down
#295That's funny -- I work in genealogy software, and loops ("being your own grandpa") happen all the time, due to data entry errors. To avoid infinite recursion, we always keep track of what records we've processed already, check whether "I've been there before", and bail out if the answer is affirmative.
Re: Why HN was down
#296Earlier quoted context omitted.
I've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have select count(distinct column_name) having count(distinct column_name) > 1; To make sure that there isn't an underlying uniqueness assumption. Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one…
This. Back in the day when I was in more of an analyst role, I ended up /having/ to hack on the live DB frequently (reasons for this were myriad). 1. Always, always make a backup just before the hack. 2. Write a small set queries like 3pt14159's to check uniqueness and other pertinent properties. 3. Write a SELECT query to show the data you are going to change. 4. Borrow the WHERE clause from 3, and write your UPDATE…
One of the things I prefer to do is to only write UPDATE statements that update a single row. For example instead of:
UPDATE line_items SET quantity = 1 WHERE quantity I'd script the following updates:
UPDATE line_items SET quantity = 1 WHERE quantity For each of the individual rows that needed to be changed. Then I have a check that I'm really updating just the rows I expect, this is especially important to me where the UPDATE involves joins, as I find this is the trickiest to get right.
Re: Why HN was down
#297So, uh, still fixing stuff in production?
Re: Why HN was down
#298Earlier quoted context omitted.
But at some point, you will become complacent, and it will taken a mistake to remind yourself again. We've all done it. I shut down an NT4 production server because I was connected via remote desktop and clicked shutdown rather than log off. This was back in the day when there was no pop-up asking for reason you want to shut down and confirmation. Luckily it was just our internal intranet server!
You were running NT4 Terminal Server Edition?
Yes, i think so. It was so long ago and it was my first programming role!
Re: Why HN was down
#299Earlier quoted context omitted.
Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.
Too bad we rarely get a postmortem on batshit insane production hackery that actually goes off without a hitch.
I would like to turn this into a poster or a t-shirt.
Re: Why HN was down
#300Earlier quoted context omitted.
I've lost count the amount of times I've done that. Also, I'm probably the top of the pops in answer replies to my own questions.
I also feel guilty when I do this, but at least the answer helps others who might have the same question.
I think stackoverflow encourages[1] this, so no need to feel guilty.
1. http://blog.stackoverflow.com/2011/07/its-ok-to-ask-and-answ...