Reminds me of NASDAQ's post from over a year ago: http://news.ycombinator.com/item?id=2928519
Good luck FogCreek.
Fog Creek is about to go down
21–30 of 193 posts
Re: Fog Creek is about to go down
#22I guess "The Cloud" got rained on in this case.
Re: Fog Creek is about to go down
#23I'm assuming their servers are in LGA11, according to Internap the basement has flooded and damaged the fuel pumps for the generators. They also said in an earlier email they have no staff on site and are urging all customers to shut down their servers. Edit: here are the emails I've received from Internap regarding LGA11 https://gist.github.com/3980482
Cut long lines to make it more readable. https://gist.github.com/3980570
Re: Fog Creek is about to go down
#24Earlier quoted context omitted.
It is movable. It would take a few days. We have another DC. We wouldn't say that if we knew it wasn't doable. EDIT: Just because it's feasible doesn't mean we will actually do it; just wanted to clarify that we weren't firing from the hip.
Wow you expect fog creek to be down for that long when it does? Why can't you just scp everything on there to your other DC? or at least just move the harddrives? Seems like it would cost less time.
Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up?
Having power and/or rack space is not the same as having servers, switches, etc. anyway.
Hopefully it will not be down that long. We'll let you know more when we know more.
Re: Fog Creek is about to go down
#25No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?
Ofcourse they have redundancy, just not cross-datacenter redundancy. And if you knew anything about cross-datacenter redundancy you'd know that cross-datacenter redundancy is something you do not decide upon lightly. Then again, having cross-datacenter backups that can easily be taken online would be a bit more professional than 'we want to physically move the servers'.
Are you kidding me? If you run big sites like FogBugz then ofcourse you have cross-datacenter redundancy. It's not complicated to host your staging site in another physical location and point the DNS records to it when things go pear-shaped.
Re: Fog Creek is about to go down
#26We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available.
Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.
Re: Fog Creek is about to go down
#27Earlier quoted context omitted.
It is movable. It would take a few days. We have another DC. We wouldn't say that if we knew it wasn't doable. EDIT: Just because it's feasible doesn't mean we will actually do it; just wanted to clarify that we weren't firing from the hip.
Wow you expect fog creek to be down for that long when it does? Why can't you just scp everything on there to your other DC? or at least just move the harddrives? Seems like it would cost less time.
I can't imagine that relocating an entire environment for a multitude of applications and services is as simple as scping things over. I'm sure there is a process wherein scp could be a step, but I don't see it being any easier/faster.
Moving the hard drives could be an option, and I believe it is sometimes done, but it assumes there are empty boxes on the other end waiting to receive the hard drives in a similar configuration to how the hard drives came. Also there's separate issues depending how many drives they are dealing with, and what redundancy is involved. If it's very few, then you might as well move the server outright. If it's very many, then there's extra human overhead (and room for error) in keeping the drives together.
Re: Fog Creek is about to go down
#28Hi guys, Joel from Fog Creek here. We have absolutely no idea what failover means, or what the hell these people are talking about with this "two datacenters in different places" bullshit. Thanks for your money.
Re: Fog Creek is about to go down
#29Earlier quoted context omitted.
Ofcourse they have redundancy, just not cross-datacenter redundancy. And if you knew anything about cross-datacenter redundancy you'd know that cross-datacenter redundancy is something you do not decide upon lightly. Then again, having cross-datacenter backups that can easily be taken online would be a bit more professional than 'we want to physically move the servers'.
Are you kidding me? If you run big sites like FogBugz then ofcourse you have cross-datacenter redundancy. It's not complicated to host your staging site in another physical location and point the DNS records to it when things go pear-shaped.
So which of your big sites have cross-datacenter redundancy? Why don't you talk about the decision process that lead to that and costs associated?
Unless you're just talking out of your arse of course and you have no experience with that sort of thing at all.
Re: Fog Creek is about to go down
#30[deleted]