Submerged fuel pumps are the same problem that Internap are experiencing. https://news.ycombinator.com/item?id=4715889
Fog Creek's services are hosted at Internap's LGA11 (75 Broad St) data center.
Fog Creek is about to go down
11–20 of 193 posts
Re: Fog Creek is about to go down
#12I'm assuming their servers are in LGA11, according to Internap the basement has flooded and damaged the fuel pumps for the generators. They also said in an earlier email they have no staff on site and are urging all customers to shut down their servers.
Edit: here are the emails I've received from Internap regarding LGA11 https://gist.github.com/3980482
Re: Fog Creek is about to go down
#13"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update.
Try not to let your fingers type cheques your datacenters can't cash...!
Re: Fog Creek is about to go down
#14There's pretty much two ways to deal with this. Either admit this is a low probability failure scenario and it isn't cost effective to have global redundancies. The outage will be resolved as soon as possible. Or, admit you failed to build a georedundant HA infrastructure and apologize with a tentative plan to build out a redundant infrastructure in a different catastrophe zone. move the servers? On what planet is se…
It is movable. It would take a few days. We have another DC. We wouldn't say that if we knew it wasn't doable.
EDIT: Just because it's feasible doesn't mean we will actually do it; just wanted to clarify that we weren't firing from the hip.
Re: Fog Creek is about to go down
#15We are beginning to bring down all Fog Creek services (FogBugz, Kiln, Trello, etc.) as our datacenter is shutting down.
fogcreekstatus.typepad.com and @fogcreekstatus on twitter will continue to have updates.
edit: All Fog Creek services have been shut down ahead of power failure. We'll update the status blog as we know more from our DC.
Re: Fog Creek is about to go down
#16Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com
Re: Fog Creek is about to go down
#17No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud.
EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?
Re: Fog Creek is about to go down
#18There's pretty much two ways to deal with this. Either admit this is a low probability failure scenario and it isn't cost effective to have global redundancies. The outage will be resolved as soon as possible. Or, admit you failed to build a georedundant HA infrastructure and apologize with a tentative plan to build out a redundant infrastructure in a different catastrophe zone. move the servers? On what planet is se…
It is movable. It would take a few days. We have another DC. We wouldn't say that if we knew it wasn't doable. EDIT: Just because it's feasible doesn't mean we will actually do it; just wanted to clarify that we weren't firing from the hip.
Wow you expect fog creek to be down for that long when it does? Why can't you just scp everything on there to your other DC? or at least just move the harddrives? Seems like it would cost less time.
Re: Fog Creek is about to go down
#19I initially read that as Fog Creek the company was soon to be going out of business.
Re: Fog Creek is about to go down
#20No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?
Ofcourse they have redundancy, just not cross-datacenter redundancy. And if you knew anything about cross-datacenter redundancy you'd know that cross-datacenter redundancy is something you do not decide upon lightly.
Then again, having cross-datacenter backups that can easily be taken online would be a bit more professional than 'we want to physically move the servers'.