61–69 of 69 posts
Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years? EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.
There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...
I'm sure you've thought of this, but now that you're up and running, isn't it possible to buy/hire a pump, run it off the electricity from the already running generators, and pump it up from street level instead, instead of walking up stairs with buckets?
Anyway, the short answer is no, it's not at all easy to pump anything up 17 stories. You're talking about 8 or more pumps and appropriate hoses curling up the staircases, making them impassable... but people need those stairs.
Earlier quoted context omitted.
Well, but it's not like AWS didn't have its share of downtime in the last couple of weeks. Back then the recommendation of the arm-chair-admins was to move to dedicated hardware you own. Oh the irony.
Not necessarily. If they only have their setup in a single AWS datacenter, and it gets hit by a hurricane, then you still have issues. The real issue is geographic redundancy, regardless of using AWS or dedicated hardware.
Earlier quoted context omitted.
Wow, that's going beyond the extra mile. I'm pretty far from NYC but if I were closer I'd give you guys a hand. In lieu of that, any possibility of a "Trello Hurricane fund" which we can donate to? Would give appreciative users like me a chance to help pitch in for a team and product which has already helped so much. Maybe it'll amount to just enough for lunch or something, but I for one would feel much better knowin…
I think there are many, many better places to donate than a profitable company who had to do a little hard labor because they didn't have a second datacenter like everyone suggests.
I'm all for donating to red cross and other charitable organizations, but that doesn't mean you can't show anyone else appreciation just because there are others with needs in the world. Give to both, hopefully more to the charitable organizations and a little less for token gifts.
Earlier quoted context omitted.
Are there any safety issues associated with hauling a bucket of diesel up a staircase? I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers?
Are there any safety issues associated with hauling a bucket of diesel up a staircase? It's flammable and slippery. Wear boots, and I suggest this is a good day to quit smoking. I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers? They are open buckets we store the cleaning supplies for the fish tank in. And…
"Trello is in the process of being moved to Amazon AWS where it will not be affected by further data center issues." I guess that's called the power of positive thinking?
I hope they move Trello back to "real" servers and not this AWS, once the datacenter is stabilized.
I think that AWS is overrated and overpriced and it's a big asset of FogCreek to still use real data centers, after all they started this thing before AWS even existed. I don't know if StackExchange or FogBugz would run so good on these AWS-servers with bad IO.
It's great though if they now have the chance to quickly overflow Trello to AWS in case of problems or high demands and buy a new correctly-priced, powerful server to put at Peer 1 a week later!
Earlier quoted context omitted.
Apologies, the link was copied from an unrelated post elsewhere and I didn't really look at the URL, no slight of hand was expected by them. I think your guess is probably wrong though, in a status post 2 days ago they listed[1] the precautions they were taking so that they could be "confident that all of our services will remain available to our customers throughout the weather", and they did not refer to another da…
Just so we're clear, I have time to reply to these because I'm stranded in LA. That's also why I have to go back and edit my posts fairly regularly when I get better information out of NY. Pretty much everyone in NYC, such as the ops team, who could answer your questions, is very much not here right now. That's why you're getting odd silences: I'm happy to answer things I know or am involved with, but not to speculat…
However, as I understand from your profile you are a coder on Kiln[1]. Are you really telling us you don't know whether there was a backup datacenter or not?
If there was, I understand you might not be able to tell us why it didn't catch the fail-over, or if not, why there wasn't one in the first place. However basic info like whether a backup existed in the first place really seems like it should be within your realm of knowledge.
Earlier quoted context omitted.
We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less. It's not like this is a well-hidden secret - head over to AWS's whitepaper section http://media.amazonwebservices.com/AWS_Cloud…
Do you know something that Netflix doesn't? They avoid EBS dependencies for a reason: http://techblog.netflix.com/2011/04/lessons-netflix-learned-... http://techblog.netflix.com/2012/10/post-mortem-of-october-2...