Well, good luck to them, both personally and in bringing it back soon.
I only wish Trello hadn't tried to reload on its own, so I could still see the screen before the shutdown. Now all I have is a blank page :(
Fog Creek is about to go down
41–50 of 193 posts
Re: Fog Creek is about to go down
#42This is a good reminder that no system is immune to failure, cloud or otherwise. Georedundancy is expensive and difficult, so it's a delicate trade-off, but engineering good physical backup systems is also difficult. Our servers are in a state far away from hurricanes, but in a state with many other natural disasters, including tornadoes, so it's hard to say if it's a good trade or not. Interesting question: why aren…
I know that Arizona is a huge place for DCs just because there aren't any natural disasters there.
There are a few of them out here in Utah that I know of, but none at the scale that they really could be. It would make a lot of sense to put some out here, I would think
Re: Fog Creek is about to go down
#43We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available. Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.
My first thought is that everyone affected is doing OK. I'm sure that everyone just wants their stuff working too, like electricity. :)
I do hope at a better time, Fogbugz can consider redundancy/failover that Stack Overflow enjoyed.
Re: Fog Creek is about to go down
#44This is a good reminder that no system is immune to failure, cloud or otherwise. Georedundancy is expensive and difficult, so it's a delicate trade-off, but engineering good physical backup systems is also difficult. Our servers are in a state far away from hurricanes, but in a state with many other natural disasters, including tornadoes, so it's hard to say if it's a good trade or not. Interesting question: why aren…
We once had space in a datacenter in Arizona, but everyone else had the same idea. We had to move out and find a new datacenter when they were at capacity and we couldn't expand. While they were building a new facility, it was over a year away and our expansion needed to happen sooner. As a final point, the added space was already being pre-reserved at a premium, and we couldn't afford the new rates vs. other areas of the U.S.
At the time, physical location wasn't a big deal, but as the company grew, and the data center overhead did as well - it became cheaper to have the core data centers closer to our operations, where our staff could be utilized. Georedunancy ultimately ended up being used for DR and minimum required service availability during major issues.
Re: Fog Creek is about to go down
#45There's pretty much two ways to deal with this. Either admit this is a low probability failure scenario and it isn't cost effective to have global redundancies. The outage will be resolved as soon as possible. Or, admit you failed to build a georedundant HA infrastructure and apologize with a tentative plan to build out a redundant infrastructure in a different catastrophe zone. move the servers? On what planet is se…
Why would moving the servers be hard? You can fit 100 terabytes of storage in a shoe-box these days. I'd be extremely surprised if you couldn't run all of FogCreek off of a single 10U blade enclosure. That would be up to 128 CPU cores; I suspect they need only a small fraction of that. On the other hand, with a 1000/Mbit uplink that they were allowed to saturate, they'd still only be able to copy out 1 terabyte in 3…
Depends on your mental model of "servers." People have different views of servers from "that nosiy hot thing under my desk" to multiple cages and dozens of racks with 1G, 10G, 40G fiber, rack switches, core switches, edge switches, terminal servers, database servers, application servers, monitoring servers, corporate boxes, .... All with various weights, accessibility, cable routing, and those damn three servers with stripped mounting screws that have been there for six years.
100 TB of high speed RAID-10 would fit in maybe 30 shoe boxes.
Then, after things are moved, you have to deal with drives that have jiggled loose, components that outright fail to work again, or things that get accidentally broken in transit.
Let's just declare an emergency federal holiday until Nov 2 so everybody can recover without dangerous heroic measures.
Re: Fog Creek is about to go down
#46Can someone explain to me how someone like Fog Creek would let an app like Trello go dark. Dont their carefully selected and perfectly screened engineers get paid gobs of money to prevent exactly this scenario from happening by having data centers in other locations replicate the one you have in your own house.
Re: Fog Creek is about to go down
#47I'm assuming their servers are in LGA11, according to Internap the basement has flooded and damaged the fuel pumps for the generators. They also said in an earlier email they have no staff on site and are urging all customers to shut down their servers. Edit: here are the emails I've received from Internap regarding LGA11 https://gist.github.com/3980482
Cut long lines to make it more readable. https://gist.github.com/3980570
> For our cloud customers, we will also being shutting down the infrastructure at this time.
That bit made me chuckle.
Re: Fog Creek is about to go down
#48"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!
Nobody was really expecting this much of Manhattan to lose power.
I would tend to disagree with you jrock. Nearly all the models talked about this storm wrecking this type of havoc.
Re: Fog Creek is about to go down
#49"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!
Nobody was really expecting this much of Manhattan to lose power.
I'm sorry, but these chaps have been around the block a few times, they're not new start-ups. They know that there is no way on earth you can guarantee (or even reasonably be sure that) a single datacenter won't fail, even under non-emergency conditions, so their customers (I'm not one) should be calling them out on why they said that all would be A-OK.
It would have been much better to say something like "We have put all reasonable preparations & precautions in place (see above), and we feel confident they will deal with most things the storm will throw at us. However please be aware that we have no fail-over datacenter available so please plan accordingly."
Re: Fog Creek is about to go down
#50Can someone explain to me how someone like Fog Creek would let an app like Trello go dark. Dont their carefully selected and perfectly screened engineers get paid gobs of money to prevent exactly this scenario from happening by having data centers in other locations replicate the one you have in your own house.
Last time I checked, Trello was a free app...