Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

181–190 of 193 posts

Re: Fog Creek is about to go down

#181

Post by Tyrannosaurs »

Supreme wrote: post #144
Tyrannosaurs wrote: post #137

Earlier quoted context omitted.

You're right, why would we want Joel to step into the field and teach us stuff when we've got you with vast knowledge and your winning manner? After all, all we get from Joel is a decade of sharing what he's worked on and why he's done stuff a particular way, in a relatively transparent manner that allows us to maybe learn stuff but importantly to put it all in a context that allows us each to make a judgement on whe…

I'm not running a blog or expecting anyone to take what they read in a comment on some site on the internet seriously. Label me however you want, it's a free internet (for now, anyway). I still find it funny how anyone can start a blog and become famous for it. Maybe I should do the same and cash in on all that buttery goodness of advertising revenue...

So are you saying that your comments shouldn't be taken seriously?

Re: Fog Creek is about to go down

#182

Post by mfrankel »

mhp wrote: post #180
mfrankel wrote: post #179

Unfortunately smaller Peer 1 companies like mine were told that power was going to be shut and we therefore brought down our servers. We're still down. I'm happy for FogCreek, and I'm generally happy with Peer 1, but I wish they would have been honest with everyone in this situation.

The issue is communication between the boots on the ground and the NOC. Earlier today, we brought our servers down too. People that stayed up (squarespace) have been up the whole time. Until I understood the whole situation, we made the same choice you did. Now that I know more and I spent the day at the DC, I realized we should get 1-2hr warning before power goes out. Peer1 should be running even after all fuel in t…

Thanks for your assessment. I actually chose Peer 1 because of the recommendation from FogCreek and I'll still thank you even after this incident.

I think it's ironic, that Peer 1 is getting accolades from Business Insider's squarespace article, while their misinforming email has hurt my firm and the small companies that use our software.

I won't hold my breath for an apology email. That's life in the big city.

Re: Fog Creek is about to go down

#183

Post by lww »

bunderbunder wrote: post #171
lww wrote: post #151

Earlier quoted context omitted.

I'm a cynic. It's going to be at LEAST a couple of days.

It's already back up. So I think that made it only two or three hours.

Yup, I'm pleasantly surprised. Although I don't envy the facilities people who are lugging jugs of diesel up 17 flights of stairs...

Re: Fog Creek is about to go down

#184

Post by modoc »

shadytrees wrote: post #93
modoc wrote: post #74

Earlier quoted context omitted.

Do you guys do off-site backups?

Yes we do. Unfortunately, that's all they are -- data backups, sans infrastructure.

Sure, but that invalidates the complaint of it being infeasible to scp out 4 TB of data before your NYC DC runs out of power. Those 4 TB of data are already out of NYC, safely in some other DC where you have hardware. You just need new servers/VMs in/near that DC to restore the backups to.

I'm not trying to 2nd guess your ops team, but the whole point in having off-site backups is to facilitate a your RTO plan in case you lose your primary DC with no warning. I guess I'd be surprised if you don't have a < 24 hour RTO plan in place. With how quickly you can get VMs and even dedicated server provisioned by many hosting providers (minutes to a couple hours), the idea of physically moving servers off-site into a new racks, with new networking, etc... seems kinda nutty...

Re: Fog Creek is about to go down

#185

Post by emmettnicholas »

hashtree wrote: post #133
spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

I feel for the guy/gal/group that draws the short straw and has to tote a 400-ish pound 55 gallon drum of diesel up 17 flights. "Whelp that lasted for 5 minutes, again!" ;)

They are carrying half-full 50-gallon barrels of diesel (source: http://forums.peer1.com/viewtopic.php?f=37&t=7532&si...).

Still, carrying 200 lb barrels up 17 flights of seawater/diesel-slicked stairs sounds ...unpleasant.

Re: Fog Creek is about to go down

#186

Post by mfrankel »

mfrankel wrote: post #182
mhp wrote: post #180

Earlier quoted context omitted.

The issue is communication between the boots on the ground and the NOC. Earlier today, we brought our servers down too. People that stayed up (squarespace) have been up the whole time. Until I understood the whole situation, we made the same choice you did. Now that I know more and I spent the day at the DC, I realized we should get 1-2hr warning before power goes out. Peer1 should be running even after all fuel in t…

Thanks for your assessment. I actually chose Peer 1 because of the recommendation from FogCreek and I'll still thank you even after this incident. I think it's ironic, that Peer 1 is getting accolades from Business Insider's squarespace article, while their misinforming email has hurt my firm and the small companies that use our software. I won't hold my breath for an apology email. That's life in the big city.

It's 7:30pm in NYC and after a few persistent support requests, we're back up.

Thanks to Michael and Joel for sharing their information and assessments with all of us.

And now that we're up, thanks to Peer 1 for doing their best in a trying situation.

Re: Fog Creek is about to go down

#187

Post by mhp »

mfrankel wrote: post #186
mfrankel wrote: post #182

Earlier quoted context omitted.

Thanks for your assessment. I actually chose Peer 1 because of the recommendation from FogCreek and I'll still thank you even after this incident. I think it's ironic, that Peer 1 is getting accolades from Business Insider's squarespace article, while their misinforming email has hurt my firm and the small companies that use our software. I won't hold my breath for an apology email. That's life in the big city.

It's 7:30pm in NYC and after a few persistent support requests, we're back up. Thanks to Michael and Joel for sharing their information and assessments with all of us. And now that we're up, thanks to Peer 1 for doing their best in a trying situation.

Send me an email at my name at my company and I'll keep you updated with what I know.

Re: Fog Creek is about to go down

#188

Post by drewcrawford »

smackfu wrote: post #53
RobAley wrote: post #13

"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!

It's not even clear what kind of prep you would do for a hurricane that ensures service will be available in a hurricane.

You might start by moving infrastructure out of the hurricane. Off the east coast would be nice, but at the very least outside of a mandatory evacuation zone [1].

Alternatively, you could design some type of system that would allow you to fail over to a geographically redundant datacenter. Joel claimed in 2007 [2] that they had such a system, and touted it as a selling point of the reliability of the hosted service. What has happened to it is probably only something that a Fog Creek engineer can tell you.

[1] http://news.ycombinator.com/item?id=4718653

[2] http://webcache.googleusercontent.com/search?q=cache:lHEK939...

Re: Fog Creek is about to go down

#189

Post by tghw »

jonny_eh wrote: post #146
tghw wrote: post #142

Earlier quoted context omitted.

That's not going to happen.

It'd be a smart way for them to finally make money off of it.

I'm no longer with the company, so I may be out of date, but the consensus was that it was just too expensive to support licensed products if it weren't necessary. The thing that people don't take into account is that there are an endless number of server configurations that can screw up the application, and for a small company, dealing with each one of those is quite expensive.

So what's the alternative? Keep it hosted-only. The downside is that, yes, outages like this happen. But I would argue that, on the whole, the overall Trello downtime has been far less than the cumulative downtime of people trying to run it themselves. Moreover, this was an extremely unusual storm. Buoys reported waves 5 times higher than anything on record. My guess is, the cost-benefit analysis is still solidly in on the side of having a hosted product in an easily accessible data center.

Re: Fog Creek is about to go down

#190

Post by tghw »

trimbo wrote: post #81

As a newish customer of Fogbugz, I'm disappointed. This storm had days, if not a week, of head's up. While I agree "no one expected Manhattan to lose power" (another comment), as someone who heads up development of a SaaS product, I would have spent most of that week planning for worst cases and recovery. I constantly think about the worst case scenarios and how long they'll take to recover, even without huge storms…

So, you agree that no one expected Manhattan to lose power. No one expected that in 2003 during the blackouts, either. And in that case, Peer1 was able to keep the servers running without disruption.

This was a monster of a storm, with unprecedented water levels. Buoys around New York reported waves 5 times higher than anything on record.

So consider it this way: would you rather the services be significantly more expensive (remember, doubling the hardware is the cheap part of it) or have the possibility of a few hours of downtime in a once in 100 years event?

Post reply on HN