Let me join in the chorus of people that are slightly miffed that I don't have access to my Trello boards. In a way it highlights again the importance of local storage and off-line accessibility for web-apps. I just checked and in the trello app on my nexus 7 I can still see (and browse) all my boards for example (the only problem is that the content is a few days old as I have not used the app recently)
Fog Creek is about to go down
111–120 of 193 posts
Re: Fog Creek is about to go down
#112Earlier quoted context omitted.
Ofcourse they have redundancy, just not cross-datacenter redundancy. And if you knew anything about cross-datacenter redundancy you'd know that cross-datacenter redundancy is something you do not decide upon lightly. Then again, having cross-datacenter backups that can easily be taken online would be a bit more professional than 'we want to physically move the servers'.
Are you kidding me? If you run big sites like FogBugz then ofcourse you have cross-datacenter redundancy. It's not complicated to host your staging site in another physical location and point the DNS records to it when things go pear-shaped.
[deleted]
Re: Fog Creek is about to go down
#113This is a good reminder that no system is immune to failure, cloud or otherwise. Georedundancy is expensive and difficult, so it's a delicate trade-off, but engineering good physical backup systems is also difficult. Our servers are in a state far away from hurricanes, but in a state with many other natural disasters, including tornadoes, so it's hard to say if it's a good trade or not. Interesting question: why aren…
Most places I've worked have had the first DC close to the office, then the second one in another country. When you're just starting up it's important to have easy physical access, and it's seldom worth migrating away from an existing DC rather than just opening up a new one.
Re: Fog Creek is about to go down
#114Curious: What kind architecture shows a 503 error when your servers are dead (like they are currently doing), but can't show an error status page? Presumably that server is not in the dead datacenter. Or is it just that something at the datacenter level is redundant?
Any kind of front-end reverse proxy could be doing that. At a complete guess, might they be using cloudflare?
Re: Fog Creek is about to go down
#115Earlier quoted context omitted.
Wow you expect fog creek to be down for that long when it does? Why can't you just scp everything on there to your other DC? or at least just move the harddrives? Seems like it would cost less time.
Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. anyway. Hopefully it will not be down that long. We'll let you know more when we know more.
Ha, I wouldn't worry too much about Kiln. I can push when the servers come back. Heck, considering how often some folks I know of push their changesets I wouldn't be surprised if a lot of them never even notice that anything happened.
FogBugz is more of a problem. Some days the "Resolve" button is my only source of job satisfaction.
Re: Fog Creek is about to go down
#116We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…
This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…
Judging by current Twitter traffic for @trello, there is a clear need for a self-hosted version.
Re: Fog Creek is about to go down
#117Earlier quoted context omitted.
Sorry, should have linked to previous evidence of the fact: http://www.codinghorror.com/blog/2006/09/has-joel-spolsky-ju...
Quoting Jeff in an attack on Joel has got to be irony yes? Whatever this post says Jeff clearly didn't share your view of Joel being an amateur and a fraud given that he went on to start a pretty successful business with him.
Zynga has (up until now) made a lot of money and they write shit code. Hell, most of the companies I've seen have made money while writing shit code. Making money indicates that a person knows how to make money. Writing good code indicates that a person knows how to write good code. Since the two are disconnected, I stand by my statement that this man is a fraud. You simply don't start a programming blog when you created a new language just to address a small concern in the project spec. Start a blog on how to make money or run a business, sure, but don't tread into a field where people are trying to produce something of quality and try to 'teach' them something.
The argument you have just presented is irrational since it's central point rests upon the fallacy of false cause.
ding!
Another satisfied customer. Next!
Re: Fog Creek is about to go down
#118I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.
Re: Fog Creek is about to go down
#119What's a little surprising to me is that as of now www.fogcreek.com returns nothing but an error message. Presumably they have control of their DNS, and could quickly throw something -- even a simple web page with a status message -- on a server at some other datacenter.
Re: Fog Creek is about to go down
#120Earlier quoted context omitted.
That means that the fuel tanks will have to be there as well or you're going to need some specialized lines/equipment to prime the pumps. How much would a 3" cylinder of diesel fuel about ten foot tall weigh? (What ever it is that would be a crap load of vacuum to produce/maintain). Then you have the weight of the tanks themselves and refilling logistics. All fun engineering problems. :)
What if you pumped it up by having a hydrostatically balanced system: instead of sucking up the fuel, pump water down to drive a piston that pushes the fuel up.
Interesting idea, I'm not an engineer so I can't really speak to that. It sounds like you'd be adding more points of failure and still have the potential of a submerged pump being and issue. Maybe something other than a piston?