Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

61–70 of 193 posts

Re: Fog Creek is about to go down

#61

Post by egeozcan »

kranner wrote: post #55
mdc wrote: post #26

We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available. Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

I totally agree. Did this thing have an export function or should I be prepared creating a scraper?

Re: Fog Creek is about to go down

#62

Post by RobAley »

smackfu wrote: post #53
RobAley wrote: post #13

"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!

It's not even clear what kind of prep you would do for a hurricane that ensures service will be available in a hurricane.

There isn't much beyond what they did, assuming you don't have another datacenter to preemptively switch load to. But what you don't do is tell your customers it will be all good. You tell them what you've done and warn them they may be downtime so they can plan accordingly.

Re: Fog Creek is about to go down

#63

Post by meaydinli »

Pwntastic wrote: post #42
calinet6 wrote: post #39

This is a good reminder that no system is immune to failure, cloud or otherwise. Georedundancy is expensive and difficult, so it's a delicate trade-off, but engineering good physical backup systems is also difficult. Our servers are in a state far away from hurricanes, but in a state with many other natural disasters, including tornadoes, so it's hard to say if it's a good trade or not. Interesting question: why aren…

I know that Arizona is a huge place for DCs just because there aren't any natural disasters there. There are a few of them out here in Utah that I know of, but none at the scale that they really could be. It would make a lot of sense to put some out here, I would think

I guess one of the most famous DCs in Utah is the NSA one: http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/

Re: Fog Creek is about to go down

#64

Post by kranner »

egeozcan wrote: post #61
kranner wrote: post #55

Earlier quoted context omitted.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

I totally agree. Did this thing have an export function or should I be prepared creating a scraper?

There is an export-to-JSON, from whose output I'm now trying to salvage my project notes.

There is also a print option, but that doesn't seem to print the back of the cards so it's pretty useless, unless I'm mistaken.

Re: Fog Creek is about to go down

#65

Post by burningion »

Supreme wrote: post #17

No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?

There's a huge difference between code you've written in your spare time, and code that exists in production.

Code that exists in production is often buggy and unwieldy, and doesn't necessarily make a lot of sense. Because when you have a product that makes money, your priorities also change.

You need to become more defensive about your maneuvers, and you have to have a real reason to justify changing code.

To commit to doing redundancy well, you need a lot of resources, and you need to have a justify diverting resources that could otherwise be used to build a better product.

There's a common misconception that you can just throw stuff at the cloud (AWS, Heroku, etc), and things will just stay up. In practice, between cacheing, database server backups, heavy writes, and crazy growth, there's a lot to deal with. It's not nearly a solved or a simple problem.

So people are probably down voting you because your opinion seems naive to them. I've personally migrated a top 80,000 global eCommerce operation, and everything broke in a million different places, and we spent 2 weeks afterwards getting things working properly again.

There's a big difference between the way things are in your head, and the way things are in the production. Don't say people don't know what they're doing because they don't have a perfect system. No system is perfect.

Re: Fog Creek is about to go down

#66

Post by smackfu »

mikeash wrote: post #54
unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

That's the first thing I thought of too. Seems like there should be a new rule that your backup generator infrastructure should be on at least the second floor.

Most basements are already waterproof (since you don't want groundwater getting in). It's generally the first floor that is the weak point, when the water gets over the basement walls. So if you think about it, it's not that hard to make the basement waterproof to a flood... just make the walls higher.

Re: Fog Creek is about to go down

#67

Post by benajnim »

egeozcan wrote: post #61
kranner wrote: post #55

Earlier quoted context omitted.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

I totally agree. Did this thing have an export function or should I be prepared creating a scraper?

There is an api: trello.com/docs/api

Re: Fog Creek is about to go down

#68

Post by hellmans »

gecko wrote: post #24
tinco wrote: post #18

Earlier quoted context omitted.

Wow you expect fog creek to be down for that long when it does? Why can't you just scp everything on there to your other DC? or at least just move the harddrives? Seems like it would cost less time.

Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. anyway. Hopefully it will not be down that long. We'll let you know more when we know more.

That 90 minutes claim is a bit disingenuous.

We're at the same data center. Internap told us that the fuel pumps were flooded and asked us to shut down at ~9:30pm last night. The generators lasted until 10:52am. So 12 hours+ of warning.

To be fair, at 9:30 they did warn that the generators would only last for 4-5 hours, but customers like us who proactively shut everything down extended that significantly.

Re: Fog Creek is about to go down

#69

Post by TillE »

jrockway wrote: post #59
jusben1369 wrote: post #48

Earlier quoted context omitted.

I would tend to disagree with you jrock. Nearly all the models talked about this storm wrecking this type of havoc.

I meant ordinary citizens, not emergency planners. Despite being told that they could be without power, many of my friends did not believe it. "How could Manhattan lose power?"

> "How could Manhattan lose power?"

Seriously? Were they not living in New York in 2003?

Re: Fog Creek is about to go down

#70

Post by Supreme »

benjaminwootton wrote: post #58
Supreme wrote: post #17

No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?

FWIW I agreed with you but downvoted because of the posting style. The decision to avoid cross data center replication was probably a carefully considered one instead of amateurish. They probably have multiple layers of redundancy in their setup and decided that the cost and overhead of cross data centre replication was not justified. In hindsight this doesn't seem like such a good decision, but I don't see how that…

Sorry, should have linked to previous evidence of the fact: http://www.codinghorror.com/blog/2006/09/has-joel-spolsky-ju...
Post reply on HN