Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trello has moved to AWS

status.fogcreek.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Trello has moved to AWS

#41

Post by andyjohnson0 »

RobAley wrote: post #29
gecko wrote: post #8

There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...

When FogBugz on Demand launched, Joel touted 2 data centers[1] , in NY and LA, "for the once-in-a-lifetime case of an entire data center blowing up". What happened to the second data center? [1] http://webcache.googleusercontent.com/search?q=cache:lHEK939...

No need to post a link to the Google cache. The original source [1] contains the same text. I don't think that there is much risk that they might ninja-edit the five year-old announcement to change the claim about redundancy.

My guess is that the second data centre was there, but failover didn't work for some reason. It would be good to know the reason, though.

[1] http://www.joelonsoftware.com/items/2007/07/09.html

Edit: Politeness

Re: Trello has moved to AWS

#42

Post by maxerickson »

grey-area wrote: post #18
gecko wrote: post #8

There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...

I'm sure you've thought of this, but now that you're up and running, isn't it possible to buy/hire a pump, run it off the electricity from the already running generators, and pump it up from street level instead, instead of walking up stairs with buckets?

They probably (only) need a couple of gallons per hour.

If they can't tie into their existing fuel infrastructure, hauling it up the ~150 to 200 feet of stairs bit by bit is probably easier than trying to source and install something temporary.

Re: Trello has moved to AWS

#43

Post by andyjohnson0 »

Fjolle wrote: post #40
mseebach2 wrote: post #34

Earlier quoted context omitted.

> They are open buckets So one guy slips and there's five gallons of diesel covering what I assume is the escape route of a NYC high rise?

It's diesel. You'd have to heat it before you would be able to ignite it. But yes :)

Ignition isn't the only risk. Diesel causes severe skin irritation. I'm not sure about the effects of inhaling fumes from split uncombusted diesel, but it can't be good for a person's lungs.

Potential hazmat incident.

Re: Trello has moved to AWS

#44

Post by maxerickson »

gecko wrote: post #10
citricsquid wrote: post #9

Not entirely related, but regarding the backup fuel situation: how long does 1 "bucket" of fuel keep the power going for and what is the cost of 1 bucket of fuel?

This information is from earlier today, so I don't know how accurate it is, but: - They burn 44 gallons per hour - The tank adjacent to the generator holds ~400 gallons - Gas is currently VERY expensive, but I don't have a real number for you at the moment.

They really need around 1/2 megawatt?

Wow.

Re: Trello has moved to AWS

#45

Post by mgkimsal »

Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years?

EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.

Re: Trello has moved to AWS

#46

Post by RobAley »

andyjohnson0 wrote: post #41
RobAley wrote: post #29

Earlier quoted context omitted.

When FogBugz on Demand launched, Joel touted 2 data centers[1] , in NY and LA, "for the once-in-a-lifetime case of an entire data center blowing up". What happened to the second data center? [1] http://webcache.googleusercontent.com/search?q=cache:lHEK939...

No need to post a link to the Google cache. The original source [1] contains the same text. I don't think that there is much risk that they might ninja-edit the five year-old announcement to change the claim about redundancy. My guess is that the second data centre was there, but failover didn't work for some reason. It would be good to know the reason, though. [1] http://www.joelonsoftware.com/items/2007/07/09.html…

Apologies, the link was copied from an unrelated post elsewhere and I didn't really look at the URL, no slight of hand was expected by them.

I think your guess is probably wrong though, in a status post 2 days ago they listed[1] the precautions they were taking so that they could be "confident that all of our services will remain available to our customers throughout the weather", and they did not refer to another data center, only "Our data center". That would be the ideal place to mention any fall-back centre.

They are also resolutely not replying to public posts on various boards questioning these issues. I am cutting them some slack in that they have a lot to attend to right now, although the fact they are managing to find time to reply to some threads (like the one in which this is posted) is a tad sniffy.

Once they're back up and running I'm sure they'll update us all on why their redundancy failed or magically wasn't there.

[1] http://status.fogcreek.com/2012/10/feelin-fine-no-expected-d...

Re: Trello has moved to AWS

#47

Post by acdha »

mgkimsal wrote: post #45

Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years? EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.

We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less.

It's not like this is a well-hidden secret - head over to AWS's whitepaper section

http://media.amazonwebservices.com/AWS_Cloud_Best_Practices....

“Be a pessimist when designing architectures in the cloud”

http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Prac...

“As the AWS web hosting architecture diagram in this paper shows, we recommend that you deploy EC2 hosts across multiple Availability Zones to make your web application more fault-tolerant.”

Re: Trello has moved to AWS

#48

Post by AdamN »

Fjolle wrote: post #40
mseebach2 wrote: post #34

Earlier quoted context omitted.

> They are open buckets So one guy slips and there's five gallons of diesel covering what I assume is the escape route of a NYC high rise?

It's diesel. You'd have to heat it before you would be able to ignite it. But yes :)

Exactly. You can't light diesel on fire with just a match. You need more heat or pressure. However, the burn from gasoline or diesel on your body is intense. I was once filling an old junker with gas when it spurted out from that tank and all over me. At first it was funny and felt nice and cold on the body. Then the burning sensation started and it took a solid 25 minutes in a cold shower to calm the sting.

Re: Trello has moved to AWS

#49

Post by mgkimsal »

acdha wrote: post #47
mgkimsal wrote: post #45

Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years? EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.

We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less. It's not like this is a well-hidden secret - head over to AWS's whitepaper section http://media.amazonwebservices.com/AWS_Cloud…

But... didn't we also have some comments before about how Amazon's own control panel systems are all in one data center? If Amazon themselves don't 'get this right', perhaps it's because it's too hard, and they need to take steps to make this easier (or indeed automatic, and charge a premium)?

Re: Trello has moved to AWS

#50

Post by lmm »

jspthrowaway wrote: post #12
gecko wrote: post #10

Earlier quoted context omitted.

This information is from earlier today, so I don't know how accurate it is, but: - They burn 44 gallons per hour - The tank adjacent to the generator holds ~400 gallons - Gas is currently VERY expensive, but I don't have a real number for you at the moment.

Diesel, on-highway in New England, is averaging $4.205/gal[1] (probably more in the city, especially more in a disaster situation). Applying your information to the above question and assuming they can get just shy of five gallons per bucket, it's probably a fair estimate that each bucket carries more than $20 of diesel fuel and accounts for seven minutes of generator time. So, about $3 per minute or 5.1 cents every…

I got a similar feeling at my last^2 job when we had a client run a superbowl ad, for which they were paying a million dollars. That works out as something like $30/millisecond.
Post reply on HN