Surviving AWS Failures with a Node.js and MongoDB Stack • Hacker News Board

Re: Surviving AWS Failures with a Node.js and MongoDB Stack

#31

Post by override » Wed, Oct 24, 2012, 10:51 PM UTC

justinsb wrote: post #15

giulianob wrote: post #12

Earlier quoted context omitted.

There were comments during the failure that AWS wasn't properly switching to use the available zones during the outage. That's what I find troubling. You are paying extra for some guaranteed availability and everyone keeps saying thats how you prevent downtime during outages. Then when the times comes it doesnt work?

If you were affected by this, I hope you got a big refund (1+ month).

I'm sure he's referring to the RDS outages, and the multi-AZ RDS's that didnt fail over properly. Many of them didnt fail over properly because the issues was EBS...where all the RDS live...

Just like many have said on here plenty of times before this outage. Dont just rely on multi-AZ to be your DR while in AWS. Leverage the other regions.

Re: Surviving AWS Failures with a Node.js and MongoDB Stack

#32

Post by Teef » Thu, Oct 25, 2012, 2:15 PM UTC

shykes wrote: post #11

veesahni wrote: post #5

Earlier quoted context omitted.

"It's also way cheaper than AWS." 3 AWS Small instances cost under $200 / mo and come with 1.7GB of RAM each. The dotCloud pricing calculator is coming up with $700 / mo for 3 mongodb instances with 1.7GB of RAM. Obviously this isn't an apples to apples comparison. But How are dotCloud instances different from AWS instances?

It's cheaper at equivalent level of best practice: * For a clean architecture you want to isolate each Mongo and node process in its own system. So you need 6 instances, not 3. * You'll need load-balancers in front of these node instances. That costs extra on AWS, and is included on dotCloud. * Did you include the cost of bandwidth and disk IO in your estimate? Those are extra on AWS, but included on dotCloud. * Moni…

Recommendation is to validate "best practice" claims. Doesn't matter what hosting solution used. Measure, measure, measure to make sure not only are you getting said claim but also that the end result meets your expectations. An example is in the past I had 7 "instances" (as shykes points out make sure they are hosted on separate nodes!) 4 of which where load balances Python web app. One of the nodes was overloaded so 1 out of 4 requests was very slow (5-10x). This was a big ajax app so initial page load would hang on the request(s) to that one instance. My point was since I had measured I could see that the node was the problem and now that I am on dedicated EC2 each node is consistant. Good luck.

Re: Surviving AWS Failures with a Node.js and MongoDB Stack

#33

Post by shykes » Thu, Oct 25, 2012, 8:15 PM UTC

Teef wrote: post #32

shykes wrote: post #11

Earlier quoted context omitted.

It's cheaper at equivalent level of best practice: * For a clean architecture you want to isolate each Mongo and node process in its own system. So you need 6 instances, not 3. * You'll need load-balancers in front of these node instances. That costs extra on AWS, and is included on dotCloud. * Did you include the cost of bandwidth and disk IO in your estimate? Those are extra on AWS, but included on dotCloud. * Moni…

Recommendation is to validate "best practice" claims. Doesn't matter what hosting solution used. Measure, measure, measure to make sure not only are you getting said claim but also that the end result meets your expectations. An example is in the past I had 7 "instances" (as shykes points out make sure they are hosted on separate nodes!) 4 of which where load balances Python web app. One of the nodes was overloaded s…

That's good advice. As the saying goes: "trust, but verify".

Regarding your performance issue - most platforms (including dotCloud) enforce ram and cpu separation between nodes, but are vulnerable to IO contention at some level. This is also true for EC2 if you use EBS: your standalone instances will almost certainly, at some point, suffer from degraded and inconsistent performance because another instance is competing for IOPS [1].

You can avoid this with the new "provisioned IOPS" volumes [2], or by skipping EBS altogether for stateless components.

[1] http://blog.scalyr.com/2012/10/16/a-systematic-look-at-ec2-i...

[2] http://aws.amazon.com/about-aws/whats-new/2012/07/31/announc...