Funny to see that this happens to everyone. A week ago, while testing some stuff to locate a low-importance bug, I erased the whole user database. Fortunately we have a good restore so the problem was solved in a few minutes, but still, cold sweat here ...
Why HN was down
51–60 of 303 posts
Re: Why HN was down
#52"But then I decided to just fix it for him by doing some surgery in the repl." I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated. "For some reason I didn't check the comments after the surgery to see if they were in the right place. " More or less my point. If this w…
> I've always found it's a good idea to not deviate.
Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him?
> More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur.
How about the opposite scenario? He has done it so many times with desired results, that he didn't bother checking?
Re: Why HN was down
#53it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.
I disagree. Very few companies would think negatively of an engineer if they made such a mistake on a non-essential, non-revenue-generating fun/research project.
How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!)
I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from those guys! They had discussion forum downtime."
Re: Why HN was down
#54Isn't it curious that the comet incident over Russia happened so close to the pass of DA 14. In the intro to the book:
http://ruby.bastardsbook.com/about/#why
is the note about surgical instruments left inside. It seems just like a coincidence that this happened so close to the switch to the new server, but I wonder if it's something deeper in the subconscious mind; the change to the new server is quite a big change (I know I feel that way when I have purchased a new computer (it feels different - even if it's running the same linux as before)) and could have upset the normal checks one has in place when tweaking things.
Re: Why HN was down
#55Hacking code in the repl without testing the new behavior. We all did that. Don't lie. Once I wanted to quick fix a "gmail.ca" to "gmail.com", which I did.. but to all the users instead of just the one mistaken. Fortunately I realized by mistake really fast ;-)
Re: Why HN was down
#56You should probably make your code robust to this sort of data corruption in the future.
Re: Why HN was down
#57"But then I decided to just fix it for him by doing some surgery in the repl." I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated. "For some reason I didn't check the comments after the surgery to see if they were in the right place. " More or less my point. If this w…
> I've always found it's a good idea to not deviate. Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him? > More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more lik…
> Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him?
This is a big difference between engineering and hacking. An engineer would never regularly do something so dangerous.
But I suspect pg isn't an engineer when he works on HN, I suspect he is a hacker, and just does whatever he wants to, whenever he wants. Which is his prerogative.
Re: Why HN was down
#58So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?
> So what do you do to avoid this in the future? It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
> It's HN... there's no SLA, there's no postmortems
I didn't mean to imply that there were. I was just curious.
> there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
Since there are multiple volunteers, I think that the site always feels important to at least one of them. I imagine that some of them have gone more than a week without giving a shit about HN, but not all of them at once. So I think there is doing things better in the future. In fact, HN keeps getting improvements behind the scenes, to keep it running, keep it interesting, and keep it from getting overrun with trolls.
Re: Why HN was down
#59Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging
A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally.
This serves several purposes:
(1) It's less insane-sounding than actually talking to an inanimate object in an open work environment.
(2) It actually feels better and forces me to think more clearly when I'm talking to an actual person -- the cognitive focus is higher when the object of conversation can actually, in theory, think and talk back (YMMV).
(3) And finally, although it does require some focus on the part of the other coder, it's not nearly as taxing to them as actually helping me solve the problem or pairing up with me.
So it's a good compromise somewhere between pair programming and talking to an actual rubber duck. Again, YMMV. Maybe I'll call it "Pair Ducking."
Re: Why HN was down
#60Do you have munin monitoring on the production HN server?
That would really make situations like this easier to debug. First, it can pinpoint exactly when something started happening, which in this situation might have helped you realize the problem was caused by your change. Secondly, in this specific situation it probably would have been easier to differentiate a situation where you are running low on memory vs this completely different situation.
As somebody who spent a lot of time professionally debugging large software systems when they were misbehaving (as a Google SRE), I can tell you that looking at graphs of many key metrics (disk IO, CPU, memory, then application specific things) was always the place to start when debugging a situation, because you can learn so many things right away. When did it start? Was it a slow buildup or an immediate thing? What is the general problem (Memory?, Disk IO?, CPU?, none of the above?)? Has a similar pattern happened in the past?
Then you can start to get fancy and plot things like "messages/minute" or something and then it becomes easy to see when issues are affecting the site performance and when they aren't.