Live data from Hacker News

Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

codinghorror.com

111–120 of 175 posts

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#111
post #7

To be fair, Atwood thought his hosting provider (CrystalTech) was backing up his system. As it turns out, their entire VM backup solution failed silently, so everyone thought the backups were being made. If anything, I'd say this is a sign not to work with CrystalTech.

I'd argue that the general idea expressed in the post below applies as much to system administration as it does to programming. Rule #1: Take responsibility.

http://74.125.93.132/search?q=cache:xGcmkwq9zUkJ:www.codingh...

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#112

Earlier quoted context omitted.

Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.

Is codinghorror really mission critical? He will restore from some other backups. Downtime on a blog is not that big a deal other than people like to hate on Jeff Atwood. Now if he lost customer credit card records or something like that, then it would be warranted.

Doesn't Jeff derive a significant portion of his personal income from the site? If so, I'd say mission critical is a good way to describe it.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#113
post #48

Earlier quoted context omitted.

I think you're thinking of Daily WTF. Jeff's blog is/was just a programming blog. He also frequently said he was the world's worst coder, so...

You're right, I often confused Jeff's blog with Daily WTF. Jeff might be self-effacing, but at the same time he also dispenses a lot of programming and software development advice.

Conferring advice does not imply a claim of authority. If I give someone directions to a coffee shop in New York, the recipient should not infer that I am an expert on New York, traffic grids, coffee, etc.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#115
post #7

To be fair, Atwood thought his hosting provider (CrystalTech) was backing up his system. As it turns out, their entire VM backup solution failed silently, so everyone thought the backups were being made. If anything, I'd say this is a sign not to work with CrystalTech.

It doesn't seem to be particularly fair to the hosting provider, really, to put their name on the failpage and to tweet about how it's half their fault. Especially after you've advocated redundant backups, implied that you have them, written about the advantages of hosting your images on S3. When a mishap reveals that you've actually done none of these things, it is more than a little disingenuous to try to emphasize your hosting provider's relatively minor role in making you look foolish.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#116
post #7

To be fair, Atwood thought his hosting provider (CrystalTech) was backing up his system. As it turns out, their entire VM backup solution failed silently, so everyone thought the backups were being made. If anything, I'd say this is a sign not to work with CrystalTech.

If you fail to audit and test your disaster recovery procedures on a regular basis, then you fail at competently maintaining your infrastructure. No excuses.

Do you also test that your airbags work properly? I don't audit and test the safety measures on most of the equipment in my house: that's what I pay the supplier for and there'll be hell to pay when they screw it up. Similarly, I don't audit and test my disaster recovery procedures, because that is what I pay my hosting partner for.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#118
post #31

Earlier quoted context omitted.

If you fail to audit and test your disaster recovery procedures on a regular basis, then you fail at competently maintaining your infrastructure. No excuses.

Yes, but I'm not sure if your comment is targeted at CrystalTech or codinghorror? You can't outsource liability. If YOU are not testing YOUR backups. YOU fail.

You can't outsource liability.

So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? I bet you haven't and that why you should stop repeating that stupid soundbite. We all 'outsource' liability all the time: we pay others to perform services for us and hold them responsible for the proper execution of those services. This includes hosting content and backing up that content.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#119
post #12

Earlier quoted context omitted.

It's not a backup if it's in the same location and managed by the same process.

Also, if you don't regularly test backups they might as well not exist.

If you regularly spend time testing backups, you should start paying someone to do that. For instance, the guys who make the backup. Who should provide that as part of the service. Which mean they are responsible for failures and can be held accountable.

Re: Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

#120
post #100
post #91

Earlier quoted context omitted.

How is it pointing fingers if your hosting provider lost your site plus their backups of your site? They were being paid to provide a service, and they failed at it in basically the worst possible way (i.e. total data loss, rather than just downtime). Sure, he should have had his own backups, but how does the fact that he could have recovered better form his host's screw up change the fact that they screwed up? That'…

He is right to be angry, but I have to disagree about calling them out while the site is still done. An analysis posted after the fact can lay out where the technical failures took place, and that is the right time to describe issues with the host. Pointing fingers in anger is very frowned upon in all organizations I work with. It implies a lack of ownership of your own products and a lack of maturity in handling you…

Point taken regarding calling them out before the situation is resolved: it would be far more respectful to allow the provider a reasonable amount of time to correct the error before explicitly attacking them. In this situation, I can understand it a little more, as recovering the backups from various internet caches might seem like a time-critical operation, and it would be difficult to ask for help with that without some explanation of what's happened. That explanation certainly could be fairly vague, though. Once the situation is resolved, though, if the resolution is unsatisfactory I think it's fair to say so.

I really took issue with your initial comment because "pointing the finger" has some connotation of assigning blame unfairly or unreasonably; I 100% agree that companies with a culture of blame are poisonous, and that people should err on the side of accepting too much responsibility rather than too little, and do their best to not pass blame on to other people.

There's certainly a line, however, across which I think it's reasonable to call someone else out. Where that line is depends on the situation and your relationship: the bar for doing it within your team is astronomically high (you should basically always deal with those things internally), within even the same company is still incredibly high (likewise), but it's lower when it comes to vendor relationships. Where you draw the line is probably different from where I draw it.

So while I agree that he should have waited, I don't think that publicly expressing his anger with his hosting provider after the fact would count as "pointing the finger" or "passing the buck" or otherwise indicative of a lack of personal responsibility; to me it would be understandable frustration and anger out of having been so dramatically let down by a third-party you were contracting with. And honestly, that sort of negative public publicity is one of the strongest checks we have on companies, be they hosting companies or retail stores or any other type of establishment.

More to the point: even if he did have backups, if they really lost his data and were unable to recover it themselves, I think he'd still be justified in outing their failure publicly after the fact. But again, you're definitely right that he should have waited and given the host a chance to resolve the issue before saying anything.

Post reply on HN