Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

131–140 of 193 posts

Re: Fog Creek is about to go down

#131

Post by CanSpice »

ShawnBird wrote: post #100
cookiecaper wrote: post #86

Earlier quoted context omitted.

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

Yeah, Oregon is known for it's viscous tsunamis and earthquakes. Also if a tropical cyclone hit Oregon it would not be a hurricane, it would be a typhoon. http://www.diffen.com/difference/Hurricane_vs_Typhoon I think the idea of the second data center is that it is far away from the first one.

If a tropical cyclone hit Oregon it would probably have started as a hurricane, not a tornado. Typhoons form west of the International Date Line, and they move predominantly west. Typhoons are pretty much guaranteed to never move east across the IDL, because of the prevailing winds.

I think you got confused about the "NE Pacific Ocean" thing under the Hurricane column. The NE Pacific Ocean is the part of the Pacific Ocean that Oregon is on.

It would technically be an extratropical cyclone, but it would likely have started as a tropical hurricane off the coast of Mexico.

Re: Fog Creek is about to go down

#132

Post by achille »

unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

Note that its not as trivial as you might think. Pumps cannot pull fuel from the 17th floor, because even if they create vacuum in the tube, atmospheric pressure can't push the fuel to the 17 floor. So the pump has to be positioned at the base, thus subject to flooding.

Re: Fog Creek is about to go down

#133

Post by hashtree »

spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

I feel for the guy/gal/group that draws the short straw and has to tote a 400-ish pound 55 gallon drum of diesel up 17 flights. "Whelp that lasted for 5 minutes, again!" ;)

Re: Fog Creek is about to go down

#134

Post by marbletiles »

spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

Given an average requirement of 500 gallons an hour? That's nearly 2 tonnes. Nobody's carrying that up 17 flights.

Re: Fog Creek is about to go down

#135

Post by dholowiski »

meaty wrote: post #104
jwr wrote: post #90

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…

This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…

Fogbugz has a self hosted version too, it's your choice which one to use.

Re: Fog Creek is about to go down

#136

Post by dbecker »

I keep hearing that this is the worst natural disaster that people (who are currently alive) have ever seen in NYC.

It's a bummer that their sites are down... but I think I can go a day without my to-do list when they have a once-in-a-lifetime natural disaster.

For perspective, it's not like they have down-time once every few months.

Re: Fog Creek is about to go down

#137

Post by Tyrannosaurs »

Supreme wrote: post #117
Tyrannosaurs wrote: post #107

Earlier quoted context omitted.

Quoting Jeff in an attack on Joel has got to be irony yes? Whatever this post says Jeff clearly didn't share your view of Joel being an amateur and a fraud given that he went on to start a pretty successful business with him.

Zynga has (up until now) made a lot of money and they write shit code. Hell, most of the companies I've seen have made money while writing shit code. Making money indicates that a person knows how to make money. Writing good code indicates that a person knows how to write good code. Since the two are disconnected, I stand by my statement that this man is a fraud. You simply don't start a programming blog when you cre…

You're right, why would we want Joel to step into the field and teach us stuff when we've got you with vast knowledge and your winning manner?

After all, all we get from Joel is a decade of sharing what he's worked on and why he's done stuff a particular way, in a relatively transparent manner that allows us to maybe learn stuff but importantly to put it all in a context that allows us each to make a judgement on whether what he says is useful / interesting to us.

By contrast with you we have the rich tapestry of an anonymous account on an internet message board, a superior manner bordering on trolling and a series of aggressively worded posts.

I don't know what I was thinking. Death to Spolsky!

Just one thing. Now that you too have taken to the internet to teach the rest of us how things should be done, if someone spots any errors in what you say it's fine to term you a fraud I take it? What's good for the goose and all.

Re: Fog Creek is about to go down

#138

Post by alkaramba »

spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

Good luck, guys!

Re: Fog Creek is about to go down

#139

Post by pc86 »

rscale wrote: post #79
pc86 wrote: post #38

Earlier quoted context omitted.

I'll be the first to admit I don't really know anything about cross-datacenter redundancy; however, I always thought that was pretty high on the list once you had SaaS products that were pulling in enough revenue to warrant full-time employees outside of the founders. What are the reasons why you would choose not to do it? Are they all financial or are there other implications?

I think the biggest argument against complex cross-DC redundancy is that it can add complexity and failure modes, not just during the emergency, but every day. As a simple example, I've seen at least a half dozen people who had issues because they thought it was as simple as throwing a mysql node into each datacenter, only to discover (much later) that the databases had become inconsistent and that failing over creat…

Great explanation, thank you.
Post reply on HN