> created a loop in the comment tree; I caused an item to be its own grandchild.
Ah, the online forum equivalent of going back in time to kill your grandfather.
Why HN was down
111–120 of 303 posts
Re: Why HN was down
#112Earlier quoted context omitted.
No that's right, but I worry about apache2 being down for potentially one or two users or bots that visit/crawl my website during a one-minute reboot. Meanwhile the big boys are down for 16 hours because they do things that any other person would have gotten a decent scolding for. Just look at the points per hour this thread is getting, if I had posted this about my website on my website people would have said I was…
I'd say this is a pretty important lesson: you don't need flawless technology and zero downtime to be popular and/or profitable. You need content worth viewing. People are more than willing to put up with technical errors if it's something they want/need. Focus on providing people what they want/need, and don't worry so much about having flawless technology until you can employ a horde of PHDs.
I would say that's a observation you just made there.
Re: Why HN was down
#113There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…
> I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc. Anyway, that's why I like sudo.
Beyond the logging, which I love, I use it for the differentiation of states. I'm just a little more attentive when I type "sudo" before something.
At work, a relatively young engineer accidentally typed a command meant for a test database server into a production window. There was a big rush to restore from backups, and there was a small amount of data loss.
One thing that came out of the retrospective, requested by the engineer in question, was the production hat. Before you opened up a connection to a production machine, you had to put on the large pirate hat. You could only put it back when you had closed the connections. I didn't really need it, but it was a great way for people to learn the necessary caution.
It also ended up being a nice exclusive lock on futzing with production, and seeing it in use led to some good discussions that otherwise might not have happened. But the main thing was developing a strong differentiation of states in everybody's heads.
Re: Why HN was down
#114Thanks for the detailed explanation.
It sounds like everything was done to fix the problem except try to figure out what the problem actually was. Why not use tools to see what the program is doing, form a hypothesis, gather data to confirm or reject the hypothesis, repeat until cause found, and then take corrective action that by this point you have high confidence will work?
I realize HN is more of a side project than a production service, but the goal is the same in both cases: to restore service quickly so you can move on to other things. It feels like a more rigorous approach would allow restoring service much faster than randomly guessing about what could be wrong and applying (costly) corrective action to see if it helps.
Besides that, in many cases (including this one), you cannot randomly guess the appropriate corrective action without finding the root cause.
Re: Why HN was down
#115There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…
Typing sudo won't save you, but using a higher-level interface will. Everyone I've ever known to change something in the database by hand, everyone at all, even on a hobby project that they know like the back of their hand, has screwed it up sooner or later. At some point the pain tells you you should stop doing that, and you create an admin tool that lets you do what you need to repeatably and safely.
Re: Why HN was down
#116The pink sombrero could have saved HN: http://www.bnj.com/cowboy-coding-pink-sombrero/
Re: Why HN was down
#117Earlier quoted context omitted.
I see where you are coming from. But I am saying this didn't happen because he did things live. This happened because he entered incorrect id making a thread its own parent(or grandparent; doesn't matter). This is the kind of mistake one would make even if you were writing proper migrations. He was doing things live isn't an issue; neither is an incorrect id. The issue is the code doesn't check for loops.
I am but an egg, I have two questions. One, if the data were held in a database, should a change like this be captured in the database logs? I am seeing more and more situations where I want these, I notice that they are by default turned off for mysql and wonder if this reflects a de facto judgment that logging slows performance more than is usually worthwhile. Two, if the data were kept in a database, wouldn't some…
Databases, at least the SQL kind, really aren't good at dealing with hierarchical data, and I don't know how you'd even begin to express that kind of constraint. I don't think a traditional database is the answer here. (If it were me, once I'd done it more than twice I'd write a "move thread" admin tool in the UI, and after I screwed it up like this I'd have a place to add such a check to).
Re: Why HN was down
#118Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging
This is also why pair programming is so great.
Re: Why HN was down
#119Earlier quoted context omitted.
There is a line from Futurama that perfectly applies ton a lot of debugging. Farnsworth: My God, is it really possible? Fry: It must be possible, it's happening. Fry: By the way, what's happening?
Is this the forward time machine episode?? I love futurama more than any man could love any tv show.
I feel like your name reflects that fact. It seems to be a reference to the Banach-Tarski duplashrinker
Re: Why HN was down
#120[deleted]