Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

91–100 of 193 posts

Re: Fog Creek is about to go down

#91

Post by tinco »

Supreme wrote: post #60
tinco wrote: post #40

Earlier quoted context omitted.

Yes, so this staging site of you has exactly the same databases as your production site? Without customer data Fogbugz and Trello are useless. This means that this simple staging site of yours needs to have all data replicated to it, which means it also needs the same hardware provisioned for it, effectively doubling your physical costs, your maintenance cost and reducing the simplicity of your architecture. Ofcourse…

Data should be backed up to staging nightly anyway . There should also be scripts in place to start this process at an arbitrary point in time and to import the data into the staging server. You do not need to match the hardware if you use cloud hosting since you can scale up whenever you want. Here's where it gets really simple. Resize the staging instance to match live. Put live into maintenance mode and begin the…

I fully agree that this is how it could, and perhaps should be done. But you assume they are already on cloud hosting, which they obviously aren't. Ofcourse this is also a choice that has to be made consciously. Especially since fogcreek has been around a lot longer than the big cloud providers.

You can look to Amazon to see that cloud architecture brings with it hidden complexity that also increases risk of downtime while you relinguish a lot of control on for example the latency and bandwidth between your nodes.

What I don't know by the way, is wether the total cost of ownership is larger for colocation or for cloud hosting.

Re: Fog Creek is about to go down

#92

Post by jemfinch »

mdc wrote: post #26

We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available. Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.

Either your engineers are underpaid or you're anticipating a larger price hike than would seem justified to gain the redundancy you need.

Re: Fog Creek is about to go down

#93

Post by shadytrees »

modoc wrote: post #74
gecko wrote: post #24

Earlier quoted context omitted.

Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. anyway. Hopefully it will not be down that long. We'll let you know more when we know more.

Do you guys do off-site backups?

Yes we do. Unfortunately, that's all they are -- data backups, sans infrastructure.

Re: Fog Creek is about to go down

#94

Post by kalininalex »

danmaz74 wrote: post #50
medinismo wrote: post #46

Can someone explain to me how someone like Fog Creek would let an app like Trello go dark. Dont their carefully selected and perfectly screened engineers get paid gobs of money to prevent exactly this scenario from happening by having data centers in other locations replicate the one you have in your own house.

Last time I checked, Trello was a free app...

Because, per Spolsky, Fog Creek wants to achieve a widespread adoption first before they start charging for it (with probably a free tier remaining). Other than that, Trello is very much a commercial app.

Fogbugz IS a commercial app, and at $25/mo (if I remember right) it's in the same category as most other commercial apps, i.e. it's not particularly cheap. It's still down.

Re: Fog Creek is about to go down

#95

Post by smackfu »

Curious: What kind architecture shows a 503 error when your servers are dead (like they are currently doing), but can't show an error status page? Presumably that server is not in the dead datacenter.

Or is it just that something at the datacenter level is redundant?

Re: Fog Creek is about to go down

#97

Post by logn »

cookiecaper wrote: post #86
df07 wrote: post #16

Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

The Midwest isn't immune to the effects of hurricanes. We've frequently lost power, sometimes up to a week, when bad hurricanes come through. They just become super storms over the Midwest and take down trees which cut power lines. And snow/ice can often take out power for up to a day.

So, I'd think that rather than place something in Ohio and New York, both which can be affected by the same weather pattern within a day or two, different coasts offers better protection.

Re: Fog Creek is about to go down

#98

Post by smackfu »

cookiecaper wrote: post #86
df07 wrote: post #16

Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

You mean in Tornado Alley?

Re: Fog Creek is about to go down

#99

Post by barrkel »

emeraldd wrote: post #75
mikeash wrote: post #54

Earlier quoted context omitted.

That's the first thing I thought of too. Seems like there should be a new rule that your backup generator infrastructure should be on at least the second floor.

That means that the fuel tanks will have to be there as well or you're going to need some specialized lines/equipment to prime the pumps. How much would a 3" cylinder of diesel fuel about ten foot tall weigh? (What ever it is that would be a crap load of vacuum to produce/maintain). Then you have the weight of the tanks themselves and refilling logistics. All fun engineering problems. :)

What if you pumped it up by having a hydrostatically balanced system: instead of sucking up the fuel, pump water down to drive a piston that pushes the fuel up.

Re: Fog Creek is about to go down

#100

Post by ShawnBird »

cookiecaper wrote: post #86
df07 wrote: post #16

Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

Yeah, Oregon is known for it's viscous tsunamis and earthquakes. Also if a tropical cyclone hit Oregon it would not be a hurricane, it would be a typhoon. http://www.diffen.com/difference/Hurricane_vs_Typhoon

I think the idea of the second data center is that it is far away from the first one.

Post reply on HN