Coding Horror and blogs.stackoverflow.com experience "100% Data Loss" • Hacker News Board

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#101

Post by mcantelon » Fri, Dec 11, 2009, 10:22 PM UTC

dnsworks wrote: post #28

ube wrote: post #13

Earlier quoted context omitted.

Well - he is human though he can have a new site now called "backup horrors". All kidding aside - this is horrible. How many people validate their backups on a regular basis? (even once a year)

Automatic validation is done through every backup (and I have my guys do a review of the logs from the nightly cron procedures). Once a month we test our restore procedures.

Ditto. Also, for the stuff I don't automatically validate I run a script to check file sizes for sanity.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#102

Post by gaius » Fri, Dec 11, 2009, 10:24 PM UTC

bad_user wrote: post #84

robryan wrote: post #78

Earlier quoted context omitted.

I'd say a lot to Jeff, not only is it a popular blog standing alone and probably generates a decent income but it's also very important to him for cross promotion.

And do you think Jeff's blog just got less popular?

He's not serving any AdWords off that page :-P

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#103

Post by mcantelon » Fri, Dec 11, 2009, 10:26 PM UTC

pyre wrote: post #99

mcantelon wrote: post #93

Earlier quoted context omitted.

Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.

The same could be said for CrystalTech

I don't think anyone's saying that CrystalTech didn't also fail.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#104

Post by gaius » Fri, Dec 11, 2009, 10:27 PM UTC

mikeryan wrote: post #10

joshwa wrote: post #8

Earlier quoted context omitted.

"ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images." http://twitter.com/codinghorror/status/6573094832

egads... methinks CrystalTech will be experiencing a mass exodus. [Also makes me wonder how to test my host's backups]

Straightforward enough. 1) Get another host, preferably one a thousand miles away from your present provider 2) Do what you need to do to get a copy of your site running there 3) Refine/automate the process and practice it 'til you can do it with your eyes closed

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#105

Post by samdk » Fri, Dec 11, 2009, 10:32 PM UTC

jrockway wrote: post #64

brk wrote: post #31

Earlier quoted context omitted.

Yes, but I'm not sure if your comment is targeted at CrystalTech or codinghorror? You can't outsource liability. If YOU are not testing YOUR backups. YOU fail.

It's a blog. How much time and money is worth spending on this? (I backup my Slicehost account with their official service. If they die and my backup dies with it, that's fine; I don't have the time or money to do anything better.)

You don't need to put much time or money into it. The bare minimum of a weekly cron-automated rsync job can be done in a few minutes, even if you've never done it before.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#106

Post by krschultz » Fri, Dec 11, 2009, 10:41 PM UTC

mcantelon wrote: post #93

nkohari wrote: post #51

Earlier quoted context omitted.

I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.

Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.

Is codinghorror really mission critical? He will restore from some other backups. Downtime on a blog is not that big a deal other than people like to hate on Jeff Atwood. Now if he lost customer credit card records or something like that, then it would be warranted.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#107

Post by diN0bot » Fri, Dec 11, 2009, 10:41 PM UTC

bad_user wrote: post #84

robryan wrote: post #78

Earlier quoted context omitted.

I'd say a lot to Jeff, not only is it a popular blog standing alone and probably generates a decent income but it's also very important to him for cross promotion.

And do you think Jeff's blog just got less popular?

his position as looking down on coding horror has somewhat moved to being part of the team. not necessarily a bad thing.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#108

Post by gorm » Fri, Dec 11, 2009, 10:49 PM UTC

mcantelon wrote: post #93

nkohari wrote: post #51

Earlier quoted context omitted.

I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.

Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.

The trick is to simulate a data loss before it happens.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#109

Post by techiferous » Fri, Dec 11, 2009, 10:52 PM UTC

In case you'd like to back up your own blog, I posted a four part series on how to do it with Amazon S3 and cron: http://techiferous.com/2009/11/getting-started-with-amazon-s...

This assumes you're running your blog on your own VPS.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#110

Post by scblock » Fri, Dec 11, 2009, 10:53 PM UTC

This has to be tough. I run a small server that includes email hosting and an image gallery site that numerous people contribute to, and I get a lot of questions or complaints when it is down (usually bad network or a brief DC power outage). I have been using SSH and rsync for a long time to pull the contents of every important directory on that server to a local Solaris server running a RAIDZ pool and time slider (so I do get incremental backup).

I didn't actually lose data like Jeff, but the datacenter the server was in decided to kill the power to the machine 2 weeks before the scheduled date (poor processes and a move to a different building) and I didn't get any notification before it happened. It took another 3 weeks for them to ship my machine back to me. Becuase I had nightly backups I was able to restore email and the photo site to a new Linode instance in a few hours. Without those backups I would have been hurting bad.