That is why some organizations don't allow adhoc data fixes to be run in production. Best practice is to backup the database, run the fix against the backup, test the fix against the backup, and all being well run the fix against production.
Why HN was down
251–260 of 303 posts
Re: Why HN was down
#252I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.
I thought it was a pretty cool error. I mean, you've got to not screw up on a lot of boring things before you can screw up this interestingly. Most failures are much more boring.
Re: Why HN was down
#253Don't worry I just figured out the totally bone-headed programming mistake I made at noon today. Time is a good mediator between skill and stress.
Re: Why HN was down
#254pg, Just wondering as to why HN isn't hosted in the cloud? (e.g. on AWS, Rackspace etc.). How do you backup all the data?
Because that would cost more?
I don't really know what the benefit of cloud hosting would be in this case.
Re: Why HN was down
#255I appreciate pg's frankness here.
Re: Why HN was down
#256Earlier quoted context omitted.
People at work are amazed when I successfully debug an issue over the phone. In reality, it amounts to 50% experience plus another 50% of Sherlock Holmes: "When you have eliminated the impossible, whatever remains, however improbable, must be the truth". Once you've identified what you're dealing with via a few strategic questions, it becomes simple quite rapidly.
Debugging is often best accomplished as a binary tree search aimed by familiarity/experience. Once you can put bounds on the search, it becomes possible to get the answer in just a few questions. Totally agree.
This is the best way to work through debugging/troubleshooting as far as I can tell, amazingly a skill many people lack, and others that just understand it intuitively without it ever thinking about it. That is one of the big divisions between hackers and everyone else in my mind.
Re: Why HN was down
#257Thanks for fixing it.
Have you considered avoiding dipping into the repl to do these kind of fixes? You don't owe any of us any sort of uptime guarantee, and you're a much better programmer than I, but it strikes me as odd that you would hack against the live server instead of create some tool that would make it so you couldn't take down the whole site when making this kind of fix...
Re: Why HN was down
#258Re: Why HN was down
#259Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging
People at work are amazed when I successfully debug an issue over the phone. In reality, it amounts to 50% experience plus another 50% of Sherlock Holmes: "When you have eliminated the impossible, whatever remains, however improbable, must be the truth". Once you've identified what you're dealing with via a few strategic questions, it becomes simple quite rapidly.
It's amazing how often I am able to fix problems by simply trying all the possible solutions--often while colleagues are saying things like "stop wasting time, it can't possibly be that." But of course often it is "that".
Re: Why HN was down
#260"I don't know if Nick succeeded, because in the middle of this I gave up and went to bed." - Not a good example to your holding companies :) What would happend if they all went to bed when something goes wrong :) Just kidding.