Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

81–90 of 193 posts

Re: Fog Creek is about to go down

#81

Post by trimbo »

As a newish customer of Fogbugz, I'm disappointed.

This storm had days, if not a week, of head's up. While I agree "no one expected Manhattan to lose power" (another comment), as someone who heads up development of a SaaS product, I would have spent most of that week planning for worst cases and recovery. I constantly think about the worst case scenarios and how long they'll take to recover, even without huge storms bearing down on the data center.

So, I'm disappointed. I really love the Fogbugz and Trello products. Now I'm in a position where I have to question whether we should depend on them.

Re: Fog Creek is about to go down

#82

Post by jrockway »

TillE wrote: post #72
jrockway wrote: post #33

Earlier quoted context omitted.

Nobody was really expecting this much of Manhattan to lose power.

It's pretty amusing to see you posting this when a couple days ago, you were accusing the media of "blowing it out of proportion". Oh hey, it turns out they were right. http://news.ycombinator.com/item?id=4706959

I never use the words that you "quoted". I said "superstorm" was a stupid term. It's a hurricane. Use a well-defined word when it's available.

(The hurricane was also predicted to weaken to a tropical storm before landfall at the time I wrote that.)

Re: Fog Creek is about to go down

#83

Post by mturmon »

jrockway wrote: post #59
jusben1369 wrote: post #48

Earlier quoted context omitted.

I would tend to disagree with you jrock. Nearly all the models talked about this storm wrecking this type of havoc.

I meant ordinary citizens, not emergency planners. Despite being told that they could be without power, many of my friends did not believe it. "How could Manhattan lose power?"

I live in California, so I know what you mean (s/weather/earthquake/). People were forgetting about events like:

http://en.wikipedia.org/wiki/Northeast_blackout_of_2003 http://en.wikipedia.org/wiki/New_York_City_blackout_of_1977 http://en.wikipedia.org/wiki/Northeast_blackout_of_1965

Humans have a hard time reasoning about events on decadal time scales.

Re: Fog Creek is about to go down

#84

Post by abarringer »

Seems their previous post[0] from last night showed a bit of overconfidence?

"Consider this the "Everything is Perfectly Fine Alarm."

Having run a few HA Datacenters I don't think that level of confidence is ever warranted.

[0] http://status.fogcreek.com/2012/10/feelin-fine-no-expected-d...

Re: Fog Creek is about to go down

#85

Post by mturmon »

TillE wrote: post #69
jrockway wrote: post #59

Earlier quoted context omitted.

I meant ordinary citizens, not emergency planners. Despite being told that they could be without power, many of my friends did not believe it. "How could Manhattan lose power?"

> "How could Manhattan lose power?" Seriously? Were they not living in New York in 2003?

Same thing happens with people who moved to LA after the Northridge quake. They don't really believe in earthquakes at a gut level.

Re: Fog Creek is about to go down

#86

Post by cookiecaper »

df07 wrote: post #16

Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com

Just curious, wouldn't it have been wiser to put the failover servers somewhere in the Midwest? It's pretty much as far away from the ocean as one can get, making tsunamis/hurricanes/etc. irrelevant, low earthquake risk, and a shorter flight from NYC. Seems a little inadvisable to place the infrastructure in two coastal areas; I guess it's probably about the local talent pool.

Re: Fog Creek is about to go down

#87

Post by dreish »

Supreme wrote: post #17

No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?

I would suggest the following mental exercise the next time you want to make a comment on HN:

Imagine you are at a dinner party at Paul Graham's house. He's there, obviously, along with several startup founders, aspiring founders, and a few established industry figures, including the person you are about to disagree with or criticize.

It will undoubtedly take more effort to figure out how to frame your criticism so that it doesn't make you a pariah, but the advantage will be that you will leave open the possibility of forming beneficial business and personal relationships.

In this case, I would try describing your own successes with building redundant services, and describe some of the other approaches you found while researching ones that you have built.

Re: Fog Creek is about to go down

#89

Post by RobAley »

darkxanthos wrote: post #77
RobAley wrote: post #49

Earlier quoted context omitted.

I'm sorry, but these chaps have been around the block a few times, they're not new start-ups. They know that there is no way on earth you can guarantee (or even reasonably be sure that) a single datacenter won't fail, even under non-emergency conditions, so their customers (I'm not one) should be calling them out on why they said that all would be A-OK. It would have been much better to say something like "We have pu…

You can be reasonably sure without being certain. They didn't guarantee anything. If you misread anything they've written as "guarantee" their services will never go down, you're blaming the wrong person. (Full disclosure: I am one of their customers)

> "we are confident that all of our services will remain available"

When you get this statement from an outfit with the pedigree and experience of Fog Creek, its as close to a guarantee as you're going to find. No misreading necessary.

At the end of the day, they could not be reasonably sure, there was a non-trivial risk that they should have been (and almost certainly were) aware of (this isn't the first time bad weather and datacenters have mixed), and they didn't communicate that to their customers.

Re: Fog Creek is about to go down

#90

Post by jwr »

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution.

When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for a cloud-hosted solution to have theirs.

Post reply on HN