Fog Creek is about to go down • Hacker News Board

Re: Fog Creek is about to go down

#161

Post by espeed » Tue, Oct 30, 2012, 6:16 PM UTC

smackfu wrote: post #98

cookiecaper wrote: post #86

Earlier quoted context omitted.

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

You mean in Tornado Alley?

Tornado Alley runs from northern Texas through Oklahoma, Kansas, Nebraska, and South Dakota.

Re: Fog Creek is about to go down

#162

Post by dredmorbius » Tue, Oct 30, 2012, 6:17 PM UTC

oconnore wrote: post #145

jwr wrote: post #90

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…

Do you think that your company could do this better at a reasonable cost? How would your company handle a scenario like this internally (an entire data center becomes unrecoverable)? If you don't have a good answer, it's simply grass-is-greener thinking.

Honestly, it's not FC vs. self-hosted, it's FC vs. an alternate product.

Preferably one not locate in a hurricane path, flood plain, earthquake zone, fire area, landslide track, or subject to political or economic instability.

Or highly redundant with tested failover paths.

All of which costs money, and still doesn't assure reliability. Look at last week's AWS EBS outage and root cause analysis: the service was brought down by its own monitoring (exacerbating an existing memory bug).

It's not easy. Sandy is the most extreme hurricane to hit NYC in a century (though the second in as many years). NYC is a sufficiently important commerce and financial hub to have excellent services and recovery capabilities, but it still isn't immune to perturbations.

Re: Fog Creek is about to go down

#163

Post by shadytrees » Tue, Oct 30, 2012, 6:17 PM UTC

theycallmemorty wrote: post #143

spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

They've since completely evacuated the building, no?

The datacenter is in Zone A, which did receive the mandatory evacuation order from the city, but we have a few people from Fog Creek onsite or nearby right now and there is at least one person from PEER 1 there right now. Roads to the area are open, life is returning.

Re: Fog Creek is about to go down

#164

Post by [deleted] » Tue, Oct 30, 2012, 6:18 PM UTC

theycallmemorty wrote: post #143

spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

They've since completely evacuated the building, no?

[deleted]

Re: Fog Creek is about to go down

#165

Post by shadytrees » Tue, Oct 30, 2012, 6:21 PM UTC

kranner wrote: post #116

meaty wrote: post #104

Earlier quoted context omitted.

This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…

Judging by current Twitter traffic for @trello, there is a clear need for a self-hosted version.

Based on my past experience working on self-hosted Kiln, I firmly believe creating a self-hosted version would cost more than it would rake in. It's different for every product and it's different for every company, though.

(Sorry for the conversation derail.)

Re: Fog Creek is about to go down

#166

Post by darkarmani » Tue, Oct 30, 2012, 6:22 PM UTC

oconnore wrote: post #145

jwr wrote: post #90

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…

Do you think that your company could do this better at a reasonable cost? How would your company handle a scenario like this internally (an entire data center becomes unrecoverable)? If you don't have a good answer, it's simply grass-is-greener thinking.

An entire data center is becoming unrecoverable, but they had 3 days of advance notice to pull drives and bring up servers somewhere else. Don't people plan to have redundant data centers?

Re: Fog Creek is about to go down

#167

Post by kevingessner » Tue, Oct 30, 2012, 6:51 PM UTC

And we're back! All Fog Creek services are back on line. Our datacenter has enough fuel for several hours and is working on getting a delivery of more. We are hoping that Kiln, FogBugz, Trello, and all our services will remain up, though things are still a bit dicey.

The details are all here: http://status.fogcreek.com/2012/10/fog-creek-services-update...

Thanks everyone for your patience!

Re: Fog Creek is about to go down

#168

Post by barrkel » Tue, Oct 30, 2012, 6:56 PM UTC

emeraldd wrote: post #120

barrkel wrote: post #99

Earlier quoted context omitted.

What if you pumped it up by having a hydrostatically balanced system: instead of sucking up the fuel, pump water down to drive a piston that pushes the fuel up.

Interesting idea, I'm not an engineer so I can't really speak to that. It sounds like you'd be adding more points of failure and still have the potential of a submerged pump being and issue. Maybe something other than a piston?

A flooded piston would be more reliable than a flooded motor IMO. Lots of reliability critical things like vehicle brakes use pistons. All you would need is enclosed tubing sufficient to withstand the internal pressure. Submersion shouldn't be an issue because the system already needs to be sealed - the pressure would cause a leak if it wasn't already sealed, and in fact flooding would reduce the probability of a leak by reducing the pressure differential.

If you keep sufficient water (up to the weight of the fuel) at the location where it's needed, the entire system needn't need a pump at all when called into action, just a tap - gravity would be sufficient. This is presuming that the fuel is kept at basement level for safety purposes, of course, otherwise you could just keep the fuel where you're storing the water. You can get by with less water and active pumping, since hydraulics are easy to turn into gearing (force multiplication) effects.

Re: Fog Creek is about to go down

#169

Post by FireBeyond » Tue, Oct 30, 2012, 7:01 PM UTC

larrys wrote: post #158

oconnore wrote: post #145

Earlier quoted context omitted.

Do you think that your company could do this better at a reasonable cost? How would your company handle a scenario like this internally (an entire data center becomes unrecoverable)? If you don't have a good answer, it's simply grass-is-greener thinking.

"reasonable cost?" The "reasonable cost" is a good point of course. Also relative to what they are able to charge and how that would change their business model. One type of customer might be willing to pay for a more robust service, others wouldn't. Take any garden variety website hosting service where the charge is under $10 per month and try to operate it giving better uptime and charging, say, $20 per month and s…

Ironic, given your comment, that FogBugz is $25-30/user/month, and aimed at non-individuals, typically, so teams and above, so a client could easily be paying Fog Creek hundreds or more a month.

Re: Fog Creek is about to go down

#170

Post by FireBeyond » Tue, Oct 30, 2012, 7:02 PM UTC

darkarmani wrote: post #166

oconnore wrote: post #145

Earlier quoted context omitted.

Do you think that your company could do this better at a reasonable cost? How would your company handle a scenario like this internally (an entire data center becomes unrecoverable)? If you don't have a good answer, it's simply grass-is-greener thinking.

An entire data center is becoming unrecoverable, but they had 3 days of advance notice to pull drives and bring up servers somewhere else. Don't people plan to have redundant data centers?

This wasn't a surprise storm, and it wasn't "more severe than expected" - this has been national news for a while. More it was "eh, we'll be alright, why pay for something and not use it, we'll just apologize later".