Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 303 posts

Re: Why HN was down

#181

Post by cientifico »

kamjam wrote: post #163
davesims wrote: post #59

Earlier quoted context omitted.

A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces m…

This is so true, probably about 90% of the time my colleagues call me over about a problem they are facing, explain in detail what the problem is and then eureka! Most of the time it would actually take me much longer to figure out the exact issue since I don't know the ins/outs and subtleties of the code but it's exactly as you say. Of course, it makes me look really good cos I just "helped" them solve their issue :…

Maybe because the solution is in asking the good questions.

Re: Why HN was down

#182

Post by irahul »

chernevik wrote: post #95
irahul wrote: post #81

Earlier quoted context omitted.

I see where you are coming from. But I am saying this didn't happen because he did things live. This happened because he entered incorrect id making a thread its own parent(or grandparent; doesn't matter). This is the kind of mistake one would make even if you were writing proper migrations. He was doing things live isn't an issue; neither is an incorrect id. The issue is the code doesn't check for loops.

I am but an egg, I have two questions. One, if the data were held in a database, should a change like this be captured in the database logs? I am seeing more and more situations where I want these, I notice that they are by default turned off for mysql and wonder if this reflects a de facto judgment that logging slows performance more than is usually worthwhile. Two, if the data were kept in a database, wouldn't some…

> I notice that they are by default turned off for mysql and wonder if this reflects a de facto judgment that logging slows performance more than is usually worthwhile.

I think it's more like your application is doing the logging already(probably; most of the frameworks do). If you really need it, turn it on yourself.

> Two, if the data were kept in a database, wouldn't something like this be prevented by a constraint preventing a comment from making itself an ancestor?

Copy pasting the table from another comment.

    create table post (id int primary_key, parent_id int references post(id), child_id int references post(id), created_at timestamp)
There isn't a simple check constraint you can place to ensure a parent's, or a grand-parent's, or a grand-grand-parent's parent_id isn't child.id You will have to write a trigger.

This isn't really a big problem to solve. pg simply overlooked this problem. Had he not, he would have checked child.created_at > parent.created_at in his mutator method. So, when you do a post.parent = some_post(assuming mutator is parent=; replace it with post.setParent or (send post set-parent some-post) or whatever), it checks if post.created_at > some_post.created_at, and then assigns post.parent_id = some_post.id

Re: Why HN was down

#183

Post by 3pt14159 »

lmm wrote: post #115
lifeisstillgood wrote: post #67

There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo ever…

Typing sudo won't save you, but using a higher-level interface will. Everyone I've ever known to change something in the database by hand, everyone at all, even on a hobby project that they know like the back of their hand, has screwed it up sooner or later. At some point the pain tells you you should stop doing that, and you create an admin tool that lets you do what you need to repeatably and safely.

I've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have

    select count(distinct column_name) having count(distinct column_name) > 1;
To make sure that there isn't an underlying uniqueness assumption.

Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one day I'd have to scramble like crazy in the middle of the night trying to figure out what I screwed up for hours on end.

I'm not saying don't build an admin tool, obviously those are needed for things like banning users, but just get in there and carefully fix the data if something is wrong.

Re: Why HN was down

#185

Post by ansible »

larrys wrote: post #39
dasil003 wrote: post #13

I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.

Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.

I agree. I've been told that with motorcycle riding, the first 10K miles are the most dangerous. This is when you've gotten out of the newbie stage, but don't yet understand your own limits nor the bike's limits.

Re: Why HN was down

#186

Post by gummadi »

irahul wrote: post #30
lucb1e wrote: post #8

Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

HN runs on plain files. He wasn't modifying database, but calling functions(I believe) in the repl to change the parent id of the thread. But that apart, even if there were an admin button to change the parent id of a thread, he would still have made the same mistake. Unless the code in question was checking for loops. In that case, repl would have worked the same.

Glad that you bought this up. If you have more knowledge regarding this, can you please explain how exactly the posts & nested comments are stored directly using flat files. How are concurrency issues handled?

Re: Why HN was down

#188

Post by pseut »

Swizec wrote: post #170
davesims wrote: post #59

Earlier quoted context omitted.

A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces m…

Working from home I tend to just write out my thoughts on a piece of paper. It works perfectly.

I send myself an email for the same purpose and set up an alias for /dev/null. It makes keeping notes a lot easier since I have the copy in sent mail and can reply to it as needed in the future. (this approach seemed less crazy before I typed it out...)

Re: Why HN was down

#189

Post by dasil003 »

hnriot wrote: post #85
dasil003 wrote: post #13

I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.

he's not legendary for his IT skills.

Low blow.

Re: Why HN was down

#190

Post by corwinstephen »

It's never what you think it is. One time, I had a memory leak in a Rails app that took me TWO WEEKS to find. In the end, it came down to me putting a line of config code in the wrong section of the config file, which for some reason created a recursive loop and caused my servers to crash about once every 30 minutes. #weak
Post reply on HN