Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

1–10 of 193 posts

Re: Fog Creek is about to go down

#5

Post by seiji »

There's pretty much two ways to deal with this. Either admit this is a low probability failure scenario and it isn't cost effective to have global redundancies. The outage will be resolved as soon as possible. Or, admit you failed to build a georedundant HA infrastructure and apologize with a tentative plan to build out a redundant infrastructure in a different catastrophe zone.

move the servers? On what planet is server infrastructure movable on a whim?

Re: Fog Creek is about to go down

#7

Post by unwind »

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running).

That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power plant. :)

Makes me glad I don't work with things that are as hard to test in the real world as these kinds of backup solutions must be. I hope they manage the refuelling, somehow.

Re: Fog Creek is about to go down

#9

Post by [deleted] »

seiji wrote: post #5

There's pretty much two ways to deal with this. Either admit this is a low probability failure scenario and it isn't cost effective to have global redundancies. The outage will be resolved as soon as possible. Or, admit you failed to build a georedundant HA infrastructure and apologize with a tentative plan to build out a redundant infrastructure in a different catastrophe zone. move the servers? On what planet is se…

[deleted]
Post reply on HN