Fog Creek is about to go down • Hacker News Board

Re: Fog Creek is about to go down

#51

Post by michaelhoffman » Tue, Oct 30, 2012, 3:15 PM UTC

almost wrote: post #29

Supreme wrote: post #25

Earlier quoted context omitted.

Are you kidding me? If you run big sites like FogBugz then ofcourse you have cross-datacenter redundancy. It's not complicated to host your staging site in another physical location and point the DNS records to it when things go pear-shaped.

So which of your big sites have cross-datacenter redundancy? Why don't you talk about the decision process that lead to that and costs associated? Unless you're just talking out of your arse of course and you have no experience with that sort of thing at all.

The relationship between willingness to opine on a topic and knowledge of that topic:

http://www.smbc-comics.com/?id=2475

Re: Fog Creek is about to go down

#52

Post by cdmoyer » Tue, Oct 30, 2012, 3:16 PM UTC

medinismo wrote: post #46

Can someone explain to me how someone like Fog Creek would let an app like Trello go dark. Dont their carefully selected and perfectly screened engineers get paid gobs of money to prevent exactly this scenario from happening by having data centers in other locations replicate the one you have in your own house.

They decide that the cost is not worth the benefit.

Excluding back-seat systems engineers on sites like this, I suspect that most of their customers will be a bit upset, but give them the benefit of the doubt and be glad to pay slightly less monthly (or nothing for Trello) and suffer a short outage.

Re: Fog Creek is about to go down

#53

Post by smackfu » Tue, Oct 30, 2012, 3:19 PM UTC

RobAley wrote: post #13

"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!

It's not even clear what kind of prep you would do for a hurricane that ensures service will be available in a hurricane.

Re: Fog Creek is about to go down

#54

Post by mikeash » Tue, Oct 30, 2012, 3:21 PM UTC

unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

That's the first thing I thought of too. Seems like there should be a new rule that your backup generator infrastructure should be on at least the second floor.

Re: Fog Creek is about to go down

#55

Post by kranner » Tue, Oct 30, 2012, 3:22 PM UTC

mdc wrote: post #26

We use Fogbugz for all our internal project tracking. The consensus among our engineers is that this downtime is understandable and we'd rather deal with it, even in a mission-important web app, than pay more every month to insure redundancy was available. Frankly this is just making us appreciate Fogbugz all the more since tracking our time without it will be a real PITA.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy.

Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative.

Take my money, Fog Creek.

Re: Fog Creek is about to go down

#56

Post by jjg » Tue, Oct 30, 2012, 3:22 PM UTC

ph0rque wrote: post #3

What about Trello, is that implied as well?

Trello is now down. We're simultaneously working on migrating to AWS, as well as physically moving the hardware as mentioned above.

Re: Fog Creek is about to go down

#57

Post by blorenz » Tue, Oct 30, 2012, 3:22 PM UTC

medinismo wrote: post #46

Can someone explain to me how someone like Fog Creek would let an app like Trello go dark. Dont their carefully selected and perfectly screened engineers get paid gobs of money to prevent exactly this scenario from happening by having data centers in other locations replicate the one you have in your own house.

Sure, the situation is not ideal. Trello is a boon to my productivity and has been a gift at being free. Spolsky has given so much to the community that I, personally, can tolerate this inconvenience to my workflow.

Re: Fog Creek is about to go down

#58

Post by benjaminwootton » Tue, Oct 30, 2012, 3:22 PM UTC

Supreme wrote: post #17

No redundancy what-so-ever? What an amateur operation. I still say Joel is a fraud. EDIT: this site is amazing - divergent opinions seem to be actively discouraged given how many "points" I've lost thanks to stating mine. Is the point of this site for all of the members to think in the same way?

FWIW I agreed with you but downvoted because of the posting style.

The decision to avoid cross data center replication was probably a carefully considered one instead of amateurish. They probably have multiple layers of redundancy in their setup and decided that the cost and overhead of cross data centre replication was not justified.

In hindsight this doesn't seem like such a good decision, but I don't see how that makes someone an amateur or a fraud.

Re: Fog Creek is about to go down

#59

Post by jrockway » Tue, Oct 30, 2012, 3:24 PM UTC

jusben1369 wrote: post #48

jrockway wrote: post #33

Earlier quoted context omitted.

Nobody was really expecting this much of Manhattan to lose power.

I would tend to disagree with you jrock. Nearly all the models talked about this storm wrecking this type of havoc.

I meant ordinary citizens, not emergency planners. Despite being told that they could be without power, many of my friends did not believe it. "How could Manhattan lose power?"

Re: Fog Creek is about to go down

#60

Post by Supreme » Tue, Oct 30, 2012, 3:24 PM UTC

tinco wrote: post #40

Supreme wrote: post #25

Earlier quoted context omitted.

Are you kidding me? If you run big sites like FogBugz then ofcourse you have cross-datacenter redundancy. It's not complicated to host your staging site in another physical location and point the DNS records to it when things go pear-shaped.

Yes, so this staging site of you has exactly the same databases as your production site? Without customer data Fogbugz and Trello are useless. This means that this simple staging site of yours needs to have all data replicated to it, which means it also needs the same hardware provisioned for it, effectively doubling your physical costs, your maintenance cost and reducing the simplicity of your architecture. Ofcourse…

Data should be backed up to staging nightly anyway. There should also be scripts in place to start this process at an arbitrary point in time and to import the data into the staging server. You do not need to match the hardware if you use cloud hosting since you can scale up whenever you want.

Here's where it gets really simple. Resize the staging instance to match live. Put live into maintenance mode and begin the data transfer to staging (with a lot of cloud providers, step #1 and #2 can be done in parallel). As soon as it finishes copying, take live down, point the DNS records at staging and wait for a few minutes. Staging is now live, with all of live's data. Problem solved. Total downtime: hardly anything compared to not being prepared. Total dataloss: none.