Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was down

news.ycombinator.com

Post reply on HN
Print view View original

281–290 of 303 posts

Re: Why HN was down

#281

Post by mkubler »

mafro wrote: post #198
3pt14159 wrote: post #183

Earlier quoted context omitted.

I've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have select count(distinct column_name) having count(distinct column_name) > 1; To make sure that there isn't an underlying uniqueness assumption. Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one…

This. Back in the day when I was in more of an analyst role, I ended up /having/ to hack on the live DB frequently (reasons for this were myriad). 1. Always, always make a backup just before the hack. 2. Write a small set queries like 3pt14159's to check uniqueness and other pertinent properties. 3. Write a SELECT query to show the data you are going to change. 4. Borrow the WHERE clause from 3, and write your UPDATE…

As a tip - Also backup your staging database and have all backups using something like Rsnapshot or maybe even in a version control system, something which does point in time backups.

I learnt this after I inherited a project which had been written by some Romanians and it was pretty horrible. There was no MVC framework, it was a hacked together mess.

Somehow the live site started using the staging database instead of the production database, both were on the same server. Every time we (the devs) pushed to staging a script would grab the latest version of the live database and overwrite (drop tables) the staging database. The assumption being that the staging database is a bit like a demo server, changes made to it are temporary and just for testing, but that it should look as similar to the main website (but updated) as possible. The production database was backed up in about 5 different ways, but the staging database wasn't backed up at all.

After about a week of vanishing books, books which authors had uploaded to the self publishing portable with descriptions and other information, we realised what was wrong. Their files stayed but their accounts and book details were wiped.

In another epic fail on the same server I later moved the root folders by running the following as root (I'd probably have been stupid and run the same command if not as root but I'd have put sudo in front of it). > cd /home//public_html/public_html > mv /* ../

I was meant to mv ./* (files from the current directory into one below cause they'd been copied across into the wrong folder. Needless to say moving the root folders such as /etc and especially /lib and /bin is a BAD idea. Although is fixable, but that's another story.

Re: Why HN was down

#282

Post by grey-area »

mafro wrote: post #198
3pt14159 wrote: post #183

Earlier quoted context omitted.

I've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have select count(distinct column_name) having count(distinct column_name) > 1; To make sure that there isn't an underlying uniqueness assumption. Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one…

This. Back in the day when I was in more of an analyst role, I ended up /having/ to hack on the live DB frequently (reasons for this were myriad). 1. Always, always make a backup just before the hack. 2. Write a small set queries like 3pt14159's to check uniqueness and other pertinent properties. 3. Write a SELECT query to show the data you are going to change. 4. Borrow the WHERE clause from 3, and write your UPDATE…

Don't you have a dev db somewhere that you can replicate the live db to? Time spent setting that up will be more than repaid by the time and stress saved when you have to do a quick fix - you can simply run your changes, check it all works on your replicated site, and then make the changes on your live db (preferably with some sort of migration tool which applies the same sql and backs up first). If you have a regular backup process you could tie into that to populate the dev database.

Even if you can't replicate the entire live db, if you can automate backup, deployment of changes and test first elsewhere it makes the entire process far less fraught.

Re: Why HN was down

#284

Post by rwallace »

prawks wrote: post #133
larrys wrote: post #33

"But then I decided to just fix it for him by doing some surgery in the repl." I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated. "For some reason I didn't check the comments after the surgery to see if they were in the right place. " More or less my point. If this w…

> I've always found it's a good idea to not deviate. > you run into potential problems that you hadn't anticipated. The second statement is no reason to live by the first. In fact, I think you'd be doing yourself a disservice by staying so comfortable. Being comfortable with the unanticipated, however, is a powerful quality to have.

That's fine, but try to become comfortable with the unanticipated on a test server, not the production server.

Re: Why HN was down

#287

Post by cranklin »

You are honest and I respect that. I'm sure many companies try to play off their downtime as something far more sophisticated when in fact, it was something too embarrassing to admit. I've certainly had my fair share of embarrassingly stupid mistakes that resulted in downtime.

Re: Why HN was down

#288

Post by furyg3 »

mafro wrote: post #198
3pt14159 wrote: post #183

Earlier quoted context omitted.

I've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have select count(distinct column_name) having count(distinct column_name) > 1; To make sure that there isn't an underlying uniqueness assumption. Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one…

This. Back in the day when I was in more of an analyst role, I ended up /having/ to hack on the live DB frequently (reasons for this were myriad). 1. Always, always make a backup just before the hack. 2. Write a small set queries like 3pt14159's to check uniqueness and other pertinent properties. 3. Write a SELECT query to show the data you are going to change. 4. Borrow the WHERE clause from 3, and write your UPDATE…

Maybe I'm old school, but shouldn't this be done in an dev or acceptance environment?

I hack on the "live" DB every day, and by live I mean i sync this DB to another environment, try it out, run it on prod.

Re: Why HN was down

#290

Post by junto »

restlessmedia wrote: post #278
bbrizzi wrote: post #276

Earlier quoted context omitted.

When I'm stuck on a problem for way too long, I start typing it out in Stack Overflow. Usually by the time I'm done describing it, I've already solved it.

I've lost count the amount of times I've done that. Also, I'm probably the top of the pops in answer replies to my own questions.

I also feel guilty when I do this, but at least the answer helps others who might have the same question.
Post reply on HN