Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

121–130 of 193 posts

Re: Fog Creek is about to go down

#121

Post by spolsky »

kranner wrote: post #55
mdc wrote: post #26

We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available. Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

Geo-redundancy is a tough engineering problem. We're building a long term solution but it's a lot of work and it's not in place today.

If this is the kind of problem that excites you, we're hiring :-)

Re: Fog Creek is about to go down

#123

Post by meaty »

kami8845 wrote: post #110
meaty wrote: post #104

Earlier quoted context omitted.

This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…

If you don't mind my asking, who is `we` in this?

We have a comms NDA which prevents me revealing the company name but we're in the financial sector and are an old fashioned "enterprise company".

Re: Fog Creek is about to go down

#124

Post by kevingessner »

smackfu wrote: post #95

Curious: What kind architecture shows a 503 error when your servers are dead (like they are currently doing), but can't show an error status page? Presumably that server is not in the dead datacenter. Or is it just that something at the datacenter level is redundant?

We've shut down all of our servers to protect data, except some of the outermost infrastructure and gateways. That 503 is coming from HAProxy, our load balancer -- it's unable to send your traffic to any of the (powered-down) servers.

Re: Fog Creek is about to go down

#125

Post by [deleted] »

spolsky wrote: post #121
kranner wrote: post #55

Earlier quoted context omitted.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

Geo-redundancy is a tough engineering problem. We're building a long term solution but it's a lot of work and it's not in place today. If this is the kind of problem that excites you, we're hiring :-)

[deleted]

Re: Fog Creek is about to go down

#127

Post by dkroy »

df07 wrote: post #16

Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com

I have been following the podcasts, and when you guys mentioned this failover for the relocation of your servers the impending hurricane didn't even come to mind. That is some amazing timing, it is a good thing you had to change data centers over there in NY. Also, it is a good thing you didn't move your servers over on a bed dolly.

Re: Fog Creek is about to go down

#128

Post by dbecker »

Supreme wrote: post #60
tinco wrote: post #40

Earlier quoted context omitted.

Yes, so this staging site of you has exactly the same databases as your production site? Without customer data Fogbugz and Trello are useless. This means that this simple staging site of yours needs to have all data replicated to it, which means it also needs the same hardware provisioned for it, effectively doubling your physical costs, your maintenance cost and reducing the simplicity of your architecture. Ofcourse…

Data should be backed up to staging nightly anyway . There should also be scripts in place to start this process at an arbitrary point in time and to import the data into the staging server. You do not need to match the hardware if you use cloud hosting since you can scale up whenever you want. Here's where it gets really simple. Resize the staging instance to match live. Put live into maintenance mode and begin the…

Why do you think they aren't doing this?

Possible explanations

1) Their engineers never thought of it

2) They considered it, and it is as simple as you think... but they don't care about uptime.

3) Implementing geographic redundancy is harder than you think given whatever other constraints or environment they face.

4) Some other explanation

#3 seems like the most likely explanation to me.

Re: Fog Creek is about to go down

#129

Post by beambot »

ShawnBird wrote: post #100
cookiecaper wrote: post #86

Earlier quoted context omitted.

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

Yeah, Oregon is known for it's viscous tsunamis and earthquakes. Also if a tropical cyclone hit Oregon it would not be a hurricane, it would be a typhoon. http://www.diffen.com/difference/Hurricane_vs_Typhoon I think the idea of the second data center is that it is far away from the first one.

FYI, according to your link: "The difference between hurricane and typhoon is that tropical cyclones in the west Pacific are called Typhoons and those in the Atlantic and east Pacific Ocean are called Hurricanes." Last I checked, Oregon is in the east Pacific, so the tropical cyclone would (indeed) be called a hurricane, yes?

Re: Fog Creek is about to go down

#130

Post by RandallBrown »

unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

The pumps have to be on the bottom floor. How else will you be able to get the fuel to them?

At least in this scenario people can carry fuel up stairs to fill the generators.

Post reply on HN