Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

codinghorror.com

Post reply on HN
Print view View original

121–130 of 175 posts

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#121

Post by joe_the_user »

nkohari wrote: post #51
dnsworks wrote: post #24

Earlier quoted context omitted.

If you fail to audit and test your disaster recovery procedures on a regular basis, then you fail at competently maintaining your infrastructure. No excuses.

I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.

Hmm,

It's a lesson on what can and can't be "just bought".

Security and backups both need some top-down hands involvement. Also US recent experience dikes and air defenses are important too.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#122

Post by ryanwaggoner »

PStamatiou wrote: post #26
leftnode wrote: post #22

1. Register an Amazon AWS S3 account - https://aws-portal.amazon.com/gp/aws/developer/registration/... 2. Download my S3 backup script (or anyone's S3 Backup script) - http://github.com/leftnode/S3-Backup 3. Set up a cron to push hourly/daily/whatever tar's of your vhost's directory to S3. Spend, like, $10 a month. Thats 30gb of storage, 30gb up and 30gb down. Now, I know that may not be a lot, but I doubt codinghorr…

I do the same thing, with a script using ruby s3sync I wrote a while ago (that I should probably update): http://paulstamatiou.com/how-to-bulletproof-server-backups-w...

I'm using your script (plus zipline and automysqlbackup) on several blogs. Thanks!

In case anyone else is interested: http://code.google.com/p/ziplinebackup/ https://sourceforge.net/projects/automysqlbackup/

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#123

Post by InclinedPlane »

nkohari wrote: post #7

To be fair, Atwood thought his hosting provider (CrystalTech) was backing up his system. As it turns out, their entire VM backup solution failed silently, so everyone thought the backups were being made. If anything, I'd say this is a sign not to work with CrystalTech.

Also to be fair the backup process Atwood uses for Stack Overflow proper (et al) appears to be much more robust (regular offsite backups, etc.)

The failures in this case appear to be:

1. Not taking the robustness of his blogs seriously enough (a LOT of people make this mistake, especially with their own content).

2. Being overly trusting of the procedures of his hosting provider. He thought he could trust his hosting provider's "backups" since they are a big company with lots of customers and he paid money for said backups, turns out he got ripped off.

3. Forgetting the maxim that you have to own your core competencies, and existence should always be a core competency.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#124

Post by viraptor »

mcantelon wrote: post #93
nkohari wrote: post #51

Earlier quoted context omitted.

I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.

Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.

Joining the chorus of other people saying "WRONG".

Reason 1: I pay for my bank account one way or another. It's bank's responsibility to keep the system running and secure, not mine. Sometimes we just can't do everything ourselves and have to rely on others.

Reason 2: Many of us host something somewhere. How many do backups? How many of us check the backups? How many do check the backups checking process? (enter infinite recursion) You have to stop at some stage. You probably don't have enough time, or your time is not worth enough to check that.

He can recover most of his blog from the caches, because it was quite popular. I bet he'll be back with an almost complete archive in less than a week.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#125

Post by brk »

Confusion wrote: post #118
brk wrote: post #31

Earlier quoted context omitted.

Yes, but I'm not sure if your comment is targeted at CrystalTech or codinghorror? You can't outsource liability. If YOU are not testing YOUR backups. YOU fail.

You can't outsource liability. So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? I bet you haven't and that why you should stop repeating that stupid soundbite. We all 'outsource' liability all the time: we pay others to perform services for us and hold them responsible for the proper execution of those services. This includes hosting content and backin…

So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits?

Nope. I have a fully paid and verified home owners insurance policy though that covers any probable loss from a toaster fire.

It's sort of like having an offsite backup of my important personal stuff.

Your example is really not a parallel to a data backup. It's difficult to fully test all of the toasters failure modes in a non-destructive way. It's easy to setup an automated backup to a remote location. Given the gmail storage limits, you could tar and gzip your files and email your gmail account with the backup data.

A better example in a "homeowner" realm is the flood pans that you can buy to put under your water heater or washing machine. You'd like to be able to trust that the manufacturer made a waterproof device, but at the same time it's cheap and easy to insure yourself against the most common failure modes.

The sound bite is not stupid. People who believe you can outsource all the messyness of keeping a website alive are continually bitten in the ass by EC2, Rackspace, etc. failures.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#126

Post by klodolph »

Confusion wrote: post #116
dnsworks wrote: post #24

Earlier quoted context omitted.

If you fail to audit and test your disaster recovery procedures on a regular basis, then you fail at competently maintaining your infrastructure. No excuses.

Do you also test that your airbags work properly? I don't audit and test the safety measures on most of the equipment in my house: that's what I pay the supplier for and there'll be hell to pay when they screw it up. Similarly, I don't audit and test my disaster recovery procedures, because that is what I pay my hosting partner for.

"Reasoning by analogy"

The people who made your car were legally required to test your model of car by crashing it into another one and making sure the airbags work. This is also too expensive to test yourself. One does not modify an airbag system at all without retesting it.

Your hosting provider has no legal requirement to test their backup system. At best, they have a contractual obligation. And they don't know how to test recovering your site, because testing whether it works is a different process for each site. Additionally, it's cheap to test backup procedures. Most people have a spare computer somewhere (maybe not in the data center), and it should only take a few hours to restore a copy of your site. Once a year. For Chris's sake, you could probably do it in the background and put a movie on.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#127

Post by klodolph »

DTrejo wrote: post #53
nkohari wrote: post #7

To be fair, Atwood thought his hosting provider (CrystalTech) was backing up his system. As it turns out, their entire VM backup solution failed silently, so everyone thought the backups were being made. If anything, I'd say this is a sign not to work with CrystalTech.

If anything, I'd say this is a sign not to work with CrystalTech. After a mistake like this, it may be a sign that they will never mess up again.

Well, I'm a bit more cynical than you. However, Jack in the Box is probably one of the least likely chains to experience an E. Coli O157:H7 outbreak (since 1993, at least).

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#128

Post by klodolph »

bad_user wrote: post #87

"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" Linus Tolvards I guess people use Google's cache and archive.org these days :)

I think it's more of an observation about which of the two are _real_ men, and less an observation about Google.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#129

Post by mcantelon »

viraptor wrote: post #124
mcantelon wrote: post #93

Earlier quoted context omitted.

Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.

Joining the chorus of other people saying "WRONG". Reason 1: I pay for my bank account one way or another. It's bank's responsibility to keep the system running and secure, not mine. Sometimes we just can't do everything ourselves and have to rely on others. Reason 2: Many of us host something somewhere. How many do backups? How many of us check the backups? How many do check the backups checking process? (enter infi…

The bank analogy is interesting and a good point, but the reality of ISPs is that they are less reliable, and less regulated, than banks.

If my income is dependent on data, I make sure it gets backed up. If it's an active project, I use my backups to build my dev environments so it's fairly obvious when a backup has failed.

Post reply on HN