31–40 of 193 posts
Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com
There's pretty much two ways to deal with this. Either admit this is a low probability failure scenario and it isn't cost effective to have global redundancies. The outage will be resolved as soon as possible. Or, admit you failed to build a georedundant HA infrastructure and apologize with a tentative plan to build out a redundant infrastructure in a different catastrophe zone. move the servers? On what planet is se…
On the other hand, with a 1000/Mbit uplink that they were allowed to saturate, they'd still only be able to copy out 1 terabyte in 3 hours.
Essential quote (literally from Networking 101): "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!
What about Trello, is that implied as well?
"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!
Nobody was really expecting this much of Manhattan to lose power.
"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!
Nobody was really expecting this much of Manhattan to lose power.
No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?
Ofcourse they have redundancy, just not cross-datacenter redundancy. And if you knew anything about cross-datacenter redundancy you'd know that cross-datacenter redundancy is something you do not decide upon lightly. Then again, having cross-datacenter backups that can easily be taken online would be a bit more professional than 'we want to physically move the servers'.
Our servers are in a state far away from hurricanes, but in a state with many other natural disasters, including tornadoes, so it's hard to say if it's a good trade or not. Interesting question: why aren't there more DCs in Utah, Wyoming, Idaho, or New Mexico? And is physical location a huge determinant in where you colo your servers?
Earlier quoted context omitted.
Ofcourse they have redundancy, just not cross-datacenter redundancy. And if you knew anything about cross-datacenter redundancy you'd know that cross-datacenter redundancy is something you do not decide upon lightly. Then again, having cross-datacenter backups that can easily be taken online would be a bit more professional than 'we want to physically move the servers'.
Are you kidding me? If you run big sites like FogBugz then ofcourse you have cross-datacenter redundancy. It's not complicated to host your staging site in another physical location and point the DNS records to it when things go pear-shaped.
What is a simple no-brainer how ever is to have offline offsite backups that can easily brought online. A best practice is to have your deployment automated in such a way that deployment to a new datacenter that already has your data should be a trivial thing.
But yeah, if you're running a tight ship something things like that go overboard without anyone noticing.
Remember the story of the 100% uptime banking software, that ran for years without ever going down, always applying the patches at runtime. Then one day a patch finally came in that required a reboot, and it was discovered that in all the years of runtime patches without reboots, it was never tested if the machine could actually still boot, and ofcourse it couldn't :)