Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fog Creek is about to go down

fogcreekstatus.typepad.com

Post reply on HN
Print view View original

141–150 of 193 posts

Re: Fog Creek is about to go down

#141

Post by alkaramba »

gecko wrote: post #24
tinco wrote: post #18

Earlier quoted context omitted.

Wow you expect fog creek to be down for that long when it does? Why can't you just scp everything on there to your other DC? or at least just move the harddrives? Seems like it would cost less time.

Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. anyway. Hopefully it will not be down that long. We'll let you know more when we know more.

>4 TB of data? I didn't realize Kiln's gotten that big already. When this whole thing is sorted out, @gecko @kevingessner - would love to see a "State of the Kiln" post and some stats!

Re: Fog Creek is about to go down

#142

Post by tghw »

kranner wrote: post #116
meaty wrote: post #104

Earlier quoted context omitted.

This is precisely why we have a self host requirement for all of our software. We did have a ton of stuff in salesforce but due to a number of problems with salesforce availability and the inevitable problem of relying on British Telecom's infrastructure monkeys, it got moved to a locally hosted dynamics CRM solution with off site transaction log shipping should the office catch fire. Cost a small fortune but there i…

Judging by current Twitter traffic for @trello, there is a clear need for a self-hosted version.

That's not going to happen.

Re: Fog Creek is about to go down

#143

Post by theycallmemorty »

spolsky wrote: post #118

I'm kind of an optimist; I believe it'll only be a matter of hours total outage. The generator is fine, the equipment is fine, the internet connectivity is fine... the only problem is getting fuel up to the generator on the 17th floor, while the fuel pumps in the basement are submerged. Someone will carry it up 17 flights if need be.

They've since completely evacuated the building, no?

Re: Fog Creek is about to go down

#144

Post by Supreme »

Tyrannosaurs wrote: post #137
Supreme wrote: post #117

Earlier quoted context omitted.

Zynga has (up until now) made a lot of money and they write shit code. Hell, most of the companies I've seen have made money while writing shit code. Making money indicates that a person knows how to make money. Writing good code indicates that a person knows how to write good code. Since the two are disconnected, I stand by my statement that this man is a fraud. You simply don't start a programming blog when you cre…

You're right, why would we want Joel to step into the field and teach us stuff when we've got you with vast knowledge and your winning manner? After all, all we get from Joel is a decade of sharing what he's worked on and why he's done stuff a particular way, in a relatively transparent manner that allows us to maybe learn stuff but importantly to put it all in a context that allows us each to make a judgement on whe…

I'm not running a blog or expecting anyone to take what they read in a comment on some site on the internet seriously.

Label me however you want, it's a free internet (for now, anyway).

I still find it funny how anyone can start a blog and become famous for it. Maybe I should do the same and cash in on all that buttery goodness of advertising revenue...

Re: Fog Creek is about to go down

#145

Post by oconnore »

jwr wrote: post #90

We depend on FogBugz (hosted) to answer our support E-mails. If the downtime is on the order of several hours, I'm fine with it, these things happen. But if (as it looks like) it is on the order of days, I'll be looking for another solution. When you offer hosted services (not cheap, mind you), you take on responsibilities. Among them are disaster recovery scenarios. We do have ours and I'm expecting any company for…

Do you think that your company could do this better at a reasonable cost? How would your company handle a scenario like this internally (an entire data center becomes unrecoverable)?

If you don't have a good answer, it's simply grass-is-greener thinking.

Re: Fog Creek is about to go down

#147

Post by bduerst »

achille wrote: post #132
unwind wrote: post #7

Too bad that the backup generator refuelling pumps have been submerged (while the generators themselves are running). That sounds like some really ... unfortunate planning of the positioning of these machines, made me think of the Daiichi incident when backup assets failed to come online because parts of the backup infrastructure were destroyed. Not as serious, of course. Fog Creek's hosting isn't a nuclear power pla…

Note that its not as trivial as you might think. Pumps cannot pull fuel from the 17th floor, because even if they create vacuum in the tube, atmospheric pressure can't push the fuel to the 17 floor. So the pump has to be positioned at the base, thus subject to flooding.

You could siphon the fuel up the tube to the pump at the time of the pump's installation, right? Through a one-way valve?

Re: Fog Creek is about to go down

#148

Post by enigmo »

spolsky wrote: post #121
kranner wrote: post #55

Earlier quoted context omitted.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

Geo-redundancy is a tough engineering problem. We're building a long term solution but it's a lot of work and it's not in place today. If this is the kind of problem that excites you, we're hiring :-)

It's particularly hard to shoehorn in after the fact. Certain development models (e.g. replicated state machines) make it much easier... mix in some magic Paxos dust and it can handle machine failures as well.

Sadly the better implementations I've used myself (or have heard about) are not publicly available. The closest thing in semi-widespread use seems to be Zookeeper, but it's more like Oracle when you really wanted SQLite (standalone service vs. library).

Re: Fog Creek is about to go down

#149

Post by darkarmani »

jrockway wrote: post #33
RobAley wrote: post #13

"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather." - yesterdays update. Try not to let your fingers type cheques your datacenters can't cash...!

Nobody was really expecting this much of Manhattan to lose power.

You mean nobody that somehow missed the media coverage expected this. Those that watched the media coverage prepared for this level of disaster.

Re: Fog Creek is about to go down

#150

Post by lww »

spolsky wrote: post #121
kranner wrote: post #55

Earlier quoted context omitted.

I don't know what Fogbugz costs, but they really should start charging for Trello now, esp. if they can use some of that revenue to add geo-redundancy. Trello is fantastic, but now I'm worried that I'm too dependent on it and I should arrange an offline alternative. Take my money, Fog Creek.

Geo-redundancy is a tough engineering problem. We're building a long term solution but it's a lot of work and it's not in place today. If this is the kind of problem that excites you, we're hiring :-)

How tough it is depends on how you're engineering your geo-redundancy. I've been doing it since 1998 and a simple active-passive solution is not as hard as some would believe but it does cost money. Active-Active is much more challenging and multi-master is obviously the ideal and the most difficult to engineer with geo-network latency. I haven't seen one solution that couldn't be staged to at least provide active-passive DR capability for a pretty reasonable price. You can even do it in EC2, RackSpace Cloud or Joyent if your feeling "cloudy".
Post reply on HN