Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

101–110 of 193 posts

Re: Fog Creek is about to go down

#101

Post by rdl »

Wow, that kind of sucks for them. I hope they consider prioritizing some kind of geographic replication after the storm is done. It adds cost and complexity (which slows down development, too), but seems like a good tradeoff when you have customers depending on it.

The geographic load balancing side is basically a solved problem (although you don't want to use only DNS-based load balancing like Route53 in most cases), but the hard part is wide area replication of databases for hot failover.

It's pretty easy to do failover if you'll accept a 5-10 minute outage, though.

Re: Fog Creek is about to go down

#102

Post by kevingessner »

modoc wrote: post #74
gecko wrote: post #24

Earlier quoted context omitted.

Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. anyway. Hopefully it will not be down that long. We'll let you know more when we know more.

Do you guys do off-site backups?

Yes, we have multiple off-site backups (cloud as well as an offsite storage DC) for all customer data. All data is still safe in NYC -- we've just brought down service to prevent problems in case of an abrupt power failure.

Re: Fog Creek is about to go down

#103

Post by roc »

jrockway wrote: post #82
TillE wrote: post #72

Earlier quoted context omitted.

It's pretty amusing to see you posting this when a couple days ago, you were accusing the media of "blowing it out of proportion". Oh hey, it turns out they were right. http://news.ycombinator.com/item?id=4706959

I never use the words that you "quoted". I said "superstorm" was a stupid term. It's a hurricane. Use a well-defined word when it's available. (The hurricane was also predicted to weaken to a tropical storm before landfall at the time I wrote that.)

Earlier yesterday it wasn't a hurricane though. And shortly after landfall it was no longer a hurricane again - even before it weakened. (cold core => post-tropical cyclone?)

I think using a general catch-all that was not a narrowly-defined technical term that didn't/wouldn't universally apply was actually a prudent and defensible thing. Given their goal of collecting all the concerns of all the stages of the storm under one umbrella.

Even if their motivation was just stupid news branding/sensationalism.

Re: Fog Creek is about to go down

#104

Post by meaty »

jwr wrote: post #90

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…

This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire.

Cost a small fortune but there is nothing more expensive than not being there for your paying customers.

Re: Fog Creek is about to go down

#105

Post by Supreme »

dreish wrote: post #87
Supreme wrote: post #17

No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?

I would suggest the following mental exercise the next time you want to make a comment on HN: Imagine you are at a dinner party at Paul Graham's house. He's there, obviously, along with several startup founders, aspiring founders, and a few established industry figures, including the person you are about to disagree with or criticize. It will undoubtedly take more effort to figure out how to frame your criticism so t…

I've outlined how we solved this problem in another comment - http://news.ycombinator.com/item?id=4717713

Incidentally, I'm not here to form relationships - personal or otherwise. The primary goal of social media sites is to indulge in procrastination while advertisers bombard us with new products, not to improve one's life. For the latter, there are books, actions and real people made of flesh and blood. This reminds me a lot of some of the people I encountered in my gaming days - they tend to forget about the context of the platform they are using.

Re: Fog Creek is about to go down

#106

Post by jpdoctor »

smackfu wrote: post #36
jrockway wrote: post #33

Earlier quoted context omitted.

Nobody was really expecting this much of Manhattan to lose power.

Well, it seems like their confidence was based on their single datacenter not going down. Which seems misplaced.

... especially since there is more than one historical example for all of Manhattan losing power. (One of those times involved looting and civil strife.) Combined with the verbiage about "once-in-a-generation storm", it is fortunate that any part of Manhattan had power.

Re: Fog Creek is about to go down

#107

Post by Tyrannosaurs »

Supreme wrote: post #70
benjaminwootton wrote: post #58

Earlier quoted context omitted.

FWIW I agreed with you but downvoted because of the posting style. The decision to avoid cross data center replication was probably a carefully considered one instead of amateurish. They probably have multiple layers of redundancy in their setup and decided that the cost and overhead of cross data centre replication was not justified. In hindsight this doesn't seem like such a good decision, but I don't see how that…

Sorry, should have linked to previous evidence of the fact: http://www.codinghorror.com/blog/2006/09/has-joel-spolsky-ju...

Quoting Jeff in an attack on Joel has got to be irony yes?

Whatever this post says Jeff clearly didn't share your view of Joel being an amateur and a fraud given that he went on to start a pretty successful business with him.

Re: Fog Creek is about to go down

#108

Post by trhtrsh »

mikeash wrote: post #54
unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

That's the first thing I thought of too. Seems like there should be a new rule that your backup generator infrastructure should be on at least the second floor.

If water is leaking in, not a crushing tidal wave, all you need is a small elevation about drainage / pump to route leaking water to another location.

Re: Fog Creek is about to go down

#109

Post by tlrobinson »

mikeash wrote: post #54
unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

That's the first thing I thought of too. Seems like there should be a new rule that your backup generator infrastructure should be on at least the second floor.

Or just make the generators work submerged?

Re: Fog Creek is about to go down

#110

Post by kami8845 »

meaty wrote: post #104
jwr wrote: post #90

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…

This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…

If you don't mind my asking, who is `we` in this?
Post reply on HN