151–160 of 193 posts
I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.
We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available. Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.
Either your engineers are underpaid or you're anticipating a larger price hike than would seem justified to gain the redundancy you need.
In my experience, what self-hosted bug-tracking systems might gain in redundancy (a few hours every time a major storm hits) they lose to context switches (which hit team morale, not merely time.)
"Hey, Bob! Can you reset the MySQL server again on the bug-tracking maze set up by Fred over 4 years ago?"
Bob obligingly does so, and then, having been interrupted in the middle of a hard problem, loses his place and can't get back up to speed by the end of day. So while you may gain four hours of butt-in-seat time, you're losing four hours of real productivity on a far more frequent basis.
I'd rather pay Fog Creek to worry about that stuff, and actually ship products.
We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…
This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…
They brought all the email server management in-house, probably cost a lot of money, took a while, and when they finally turned it on... half the company was without reliable email for about a month.
Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com
Thank you for this discussion!
I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.
Sorry for what you are going through obviously. One thing I have found though in disaster planning is that it pays to be a pessimist. While worrying certainly doesn't help once you have a problem to solve doing so in advance helps you anticipate things that you need to take into consideration when planning.
1) Like any prepared person, I got all my data out of the east coast before this whole hurricane thing. If someone from Fog Creek hooked me up with some emergency licenses while they got their stuff sorted out, we'd be fine. Actually, this would be a good time to switch to self-hosted.
2) Second, while I was doing our hurricane prep, I ran into this blog post from Joel:
> Copies of the database backups are maintained in both cities, and each city serves as a warm backup for the other. If the New York data center goes completely south, we’ll wait a while to make sure it’s not coming back up, and then we’ll start changing the DNS records and start bringing up our customers on the warm backup in Los Angeles. It’s not an instantaneous failover, since customers will have to wait for two things: we’ll have to decide that a data center is really gone, not just temporarily offline, and they’ll have to wait up to 15 minutes for the DNS changes to propagate. Still, this is for the once-in-a-lifetime case of an entire data center blowing up
http://webcache.googleusercontent.com/search?q=cache:lHEK939...
Obviously this was written in 2007, but they claim to be geographically redundant and have geographic backups that are "never more than 15 minutes behind". Presumably things haven't deteriorated since then.
Earlier quoted context omitted.
Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.
The Midwest isn't immune to the effects of hurricanes. We've frequently lost power, sometimes up to a week, when bad hurricanes come through. They just become super storms over the Midwest and take down trees which cut power lines. And snow/ice can often take out power for up to a day. So, I'd think that rather than place something in Ohio and New York, both which can be affected by the same weather pattern within a…
I lived in KC for some years and we'd occasionally get remnants of gulf hurricanes, but they'd just be severe storm systems that would pass through. We rarely lost power, but if we did, it was never for protracted time periods, and generators in data centers should easily be able to deal with power loss from severe storms.
Tornadoes are pretty much the least threatening natural disaster out there, as their area of effect is usually small and their duration is usually short, so I think "tornado alley" is actually a fine place for a data center meteorlogically.
We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…
Do you think that your company could do this better at a reasonable cost? How would your company handle a scenario like this internally (an entire data center becomes unrecoverable)? If you don't have a good answer, it's simply grass-is-greener thinking.
The "reasonable cost" is a good point of course. Also relative to what they are able to charge and how that would change their business model. One type of customer might be willing to pay for a more robust service, others wouldn't. Take any garden variety website hosting service where the charge is under $10 per month and try to operate it giving better uptime and charging, say, $20 per month and see your customer base vanish. People expect it to work 24x7 but aren't willing to pay for it. They would rather take their chances.
That said one thing they might be able to do at a "reasonable" cost is simply spread their customer base over multiple data centers. So the failure of one would only bring down a smaller percentage of their customers.
Earlier quoted context omitted.
Note that its not as trivial as you might think. Pumps cannot pull fuel from the 17th floor, because even if they create vacuum in the tube, atmospheric pressure can't push the fuel to the 17 floor. So the pump has to be positioned at the base, thus subject to flooding.
You could siphon the fuel up the tube to the pump at the time of the pump's installation, right? Through a one-way valve?
So a couple of things: 1) Like any prepared person, I got all my data out of the east coast before this whole hurricane thing. If someone from Fog Creek hooked me up with some emergency licenses while they got their stuff sorted out, we'd be fine. Actually, this would be a good time to switch to self-hosted. 2) Second, while I was doing our hurricane prep, I ran into this blog post from Joel: > Copies of the database…
Not sure I see the need for a propagation delay if customers can be pointed to the new site by simply using domainbackup.com instead of domain.com (in other words completely separate dns as well as a completely different domain (even through a completely separate registrar) to a site hosted elsewhere. They can know this in advance of course.