Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trello has moved to AWS

status.fogcreek.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Trello has moved to AWS

#51

Post by andyjohnson0 »

RobAley wrote: post #46
andyjohnson0 wrote: post #41

Earlier quoted context omitted.

No need to post a link to the Google cache. The original source [1] contains the same text. I don't think that there is much risk that they might ninja-edit the five year-old announcement to change the claim about redundancy. My guess is that the second data centre was there, but failover didn't work for some reason. It would be good to know the reason, though. [1] http://www.joelonsoftware.com/items/2007/07/09.html…

Apologies, the link was copied from an unrelated post elsewhere and I didn't really look at the URL, no slight of hand was expected by them. I think your guess is probably wrong though, in a status post 2 days ago they listed[1] the precautions they were taking so that they could be "confident that all of our services will remain available to our customers throughout the weather", and they did not refer to another da…

No need for apologies. My comment was unnecessarily accusatory. I've edited it accordingly.

Re their status posts: you may well be right. Time will tell I guess.

Re: Trello has moved to AWS

#52

Post by fr0sty »

gecko wrote: post #10
citricsquid wrote: post #9

Not entirely related, but regarding the backup fuel situation: how long does 1 "bucket" of fuel keep the power going for and what is the cost of 1 bucket of fuel?

This information is from earlier today, so I don't know how accurate it is, but: - They burn 44 gallons per hour - The tank adjacent to the generator holds ~400 gallons - Gas is currently VERY expensive, but I don't have a real number for you at the moment.

Nit-pick: These generators burn diesel not, gas.

And to the poster below: 500kw generator is not unreasonable for a fully populated datacenter. That is power and cooling, remember.

Re: Trello has moved to AWS

#53

Post by TylerE »

andyjohnson0 wrote: post #43
Fjolle wrote: post #40

Earlier quoted context omitted.

It's diesel. You'd have to heat it before you would be able to ignite it. But yes :)

Ignition isn't the only risk. Diesel causes severe skin irritation. I'm not sure about the effects of inhaling fumes from split uncombusted diesel, but it can't be good for a person's lungs. Potential hazmat incident.

Diesel doesn't vaporize to any significant degree at temperatures experienced on Earth.

Re: Trello has moved to AWS

#54

Post by joezydeco »

rdl wrote: post #17

"Trello is in the process of being moved to Amazon AWS where it will not be affected by further data center issues." I guess that's called the power of positive thinking?

..or the power of a really long night with very sore muscles.

Re: Trello has moved to AWS

#55

Post by jimwalsh »

mgkimsal wrote: post #49
acdha wrote: post #47

Earlier quoted context omitted.

We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less. It's not like this is a well-hidden secret - head over to AWS's whitepaper section http://media.amazonwebservices.com/AWS_Cloud…

But... didn't we also have some comments before about how Amazon's own control panel systems are all in one data center? If Amazon themselves don't 'get this right', perhaps it's because it's too hard, and they need to take steps to make this easier (or indeed automatic, and charge a premium)?

Those comments were in the wrong, and they were posted because parts of the control panel stopped working during the last outage. This was actually due to the throttling of the API, as mentioned in their write up.

AWS is resilient enough so long as you dont park all of your environment in the same AZ/Region.

Re: Trello has moved to AWS

#56

Post by griffordson »

acdha wrote: post #47
mgkimsal wrote: post #45

Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years? EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.

We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less. It's not like this is a well-hidden secret - head over to AWS's whitepaper section http://media.amazonwebservices.com/AWS_Cloud…

Do you know something that Netflix doesn't? They avoid EBS dependencies for a reason:

http://techblog.netflix.com/2011/04/lessons-netflix-learned-...

http://techblog.netflix.com/2012/10/post-mortem-of-october-2...

Re: Trello has moved to AWS

#57

Post by lsllc »

jspthrowaway wrote: post #12
gecko wrote: post #10

Earlier quoted context omitted.

This information is from earlier today, so I don't know how accurate it is, but: - They burn 44 gallons per hour - The tank adjacent to the generator holds ~400 gallons - Gas is currently VERY expensive, but I don't have a real number for you at the moment.

Diesel, on-highway in New England, is averaging $4.205/gal[1] (probably more in the city, especially more in a disaster situation). Applying your information to the above question and assuming they can get just shy of five gallons per bucket, it's probably a fair estimate that each bucket carries more than $20 of diesel fuel and accounts for seven minutes of generator time. So, about $3 per minute or 5.1 cents every…

FYI: #2 heating oil and diesel are pretty much the same thing, the only differences being dyes in the heating oil (since it's often taxed differently) and additives in diesel (for example in the winter to stop it waxing up in the cold). Not sure about sulfur content differences when comparing #2 to the low sulfur diesels you see, but I doubt a few hours of burning higher sulfur fuel will do anything.

Also, #2 heating oil is cheaper than diesel in New England, it's about $3.70/gal.

Re: Trello has moved to AWS

#58

Post by gecko »

RobAley wrote: post #46
andyjohnson0 wrote: post #41

Earlier quoted context omitted.

No need to post a link to the Google cache. The original source [1] contains the same text. I don't think that there is much risk that they might ninja-edit the five year-old announcement to change the claim about redundancy. My guess is that the second data centre was there, but failover didn't work for some reason. It would be good to know the reason, though. [1] http://www.joelonsoftware.com/items/2007/07/09.html…

Apologies, the link was copied from an unrelated post elsewhere and I didn't really look at the URL, no slight of hand was expected by them. I think your guess is probably wrong though, in a status post 2 days ago they listed[1] the precautions they were taking so that they could be "confident that all of our services will remain available to our customers throughout the weather", and they did not refer to another da…

Just so we're clear, I have time to reply to these because I'm stranded in LA. That's also why I have to go back and edit my posts fairly regularly when I get better information out of NY. Pretty much everyone in NYC, such as the ops team, who could answer your questions, is very much not here right now. That's why you're getting odd silences: I'm happy to answer things I know or am involved with, but not to speculate.

Re: Trello has moved to AWS

#59

Post by pyre »

Xylakant wrote: post #28
zwily wrote: post #26

Earlier quoted context omitted.

You do know that AWS did not go down during Sandy, right? /me knocks on wood

Well, but it's not like AWS didn't have its share of downtime in the last couple of weeks. Back then the recommendation of the arm-chair-admins was to move to dedicated hardware you own. Oh the irony.

Not necessarily. If they only have their setup in a single AWS datacenter, and it gets hit by a hurricane, then you still have issues.

The real issue is geographic redundancy, regardless of using AWS or dedicated hardware.

Re: Trello has moved to AWS

#60

Post by gecko »

grey-area wrote: post #18
gecko wrote: post #8

There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...

I'm sure you've thought of this, but now that you're up and running, isn't it possible to buy/hire a pump, run it off the electricity from the already running generators, and pump it up from street level instead, instead of walking up stairs with buckets?

They've thought of that. Kind of fun to find new sets of industrial pumps post-hurricane though.
Post reply on HN