Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

Post reply on HN
Print view View original

71–80 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#71

Post by bob1029 »

matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

For us this is exactly the correct approach. We could have spent millions of dollars and thousands of man hours hardening things to be resilient to single region outages. But for what? We aren't GE or Google. If our conference line goes down for 2-3 hours per year because we don't have apocalypse-proof infrastructure, literally nothing bad happens to our business. In this exact outage we are discussing, all of my coworkers are at home having breakfast with their families and doing various weekend activities. No one but me will know there was even a problem until I log into the AWS console and find the alerts. Worst case, I have to reboot or restore a few affected instances on Tuesday morning.

It seems like a lot of businesses are chasing this ideal of perfect and end up much worse off than if they had just stuck whatever application on a single server in a semi-reliable part of the world.

Re: AWS EC2/RDS Outage in us-east-1

#72

Post by jacques_chester »

sdrothrock wrote: post #32

Amazon JUST had an ec2/RDS failure in one AZ in Tokyo last week; the cause was a bug in their HVAC that led to overheating. I wonder if this is similar or just coincidental. https://aws.amazon.com/jp/message/56489/

US-East-1 was the first, IIRC. Lots of early adopters have grown significantly but have various hardcoded assumptions about running there.

Re: AWS EC2/RDS Outage in us-east-1

#73

Post by cle »

jen_h wrote: post #70
matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

An occasional outage is sometimes good for an app (depending, of course, on how mission-critical it is): 1. People don't realize how much they love and depend on you until you're gone. 2. Keeps you on your toes, it's easy to get complacent when everything just runs along happily for months and years on end. I do wish there was a way to train users that millions of them reloading constantly as service ramps back up do…

Yes! If you haven’t had an operational issue with a service in a while, you should force one on a testing to stack to make sure it fails like you expect, you can recover gracefully, your monitors work, etc. (lots of folks call these “game days”). A service that hasn’t had an ops issue in a long time is a ticking time bomb: when it does fail, nobody will be very familiar with it, and environmental assumptions might have changed, which could result in taking a much longer time to recover.

When I design services these days, I try to design them so these failure scenarios are constantly exercised. Eg if I care about multi-AZ resiliency, I try to design it so that it’s forced to fail over to other AZs all the time. Or at the least, write tests for the scenario. Exceptional behavior or code paths are dangerous.

Re: AWS EC2/RDS Outage in us-east-1

#74

Post by herostratus101 »

Matthias247 wrote: post #51
bdcravens wrote: post #18

Earlier quoted context omitted.

Also, it is one of the regions that gets new features first, which makes me wonder if it contributes to lower stability.

This is not true. The region where new software is deployed first is different team by team (or service by service).

us-east-1 is the most likely to have any given new service first.

Re: AWS EC2/RDS Outage in us-east-1

#75

Post by manojlds »

matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you aren't multi AZ etc ( of course YMMV based on sensitivity of your business)

Re: AWS EC2/RDS Outage in us-east-1

#76

Post by kortilla »

bjterry wrote: post #44
xyst wrote: post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

This outage is only affecting a single availability zone, so taking on the complexity of multiple cloud providers would not be necessary to be resilient against it. AWS best practices would already have covered you.

AWS best practices involve using AWS for everything, which is not actually a best practice.

Re: AWS EC2/RDS Outage in us-east-1

#77

Post by y0y »

riffic wrote: post #59
tdurden wrote: post #19

Earlier quoted context omitted.

Zone designations are account specific; zone D for you is not zone D for me

The affected AZ appears to be use1-az6. You can map "your" AZ name (us-east-1c, us-east-1d, etc.) to the actual, canonical name of the AZ in the 'Subnets' tab on the VPC console.

What makes you say use1-az6 is the culprit? I only ask because none of our workloads in az6 have experienced any issues. ....yet. We run critical workloads across 3 AZs thankfully, but still.

Re: AWS EC2/RDS Outage in us-east-1

#79

Post by jen_h »

manojlds wrote: post #75
matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you aren't multi AZ etc ( of course YMMV based on sensitivity of your business)

Yep. I call it the "AWS Chicken Pox Party."

We got lucky this time, RDS, ELB, EC2 and Lightsail instances all in US-East-1 across multiple accounts and no issues (knock-on-wood and understand that we'll get it the next time). Especially happy as I had a two-day running neural net training job running and it's still going, that would have been depressing. Phew.

Re: AWS EC2/RDS Outage in us-east-1

#80

Post by bdcravens »

matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

Indeed. Our company's core business occurs in batch, background processing, that needn't be real time. If it doesn't run right now, there's literally no damage to the business or our customers if it runs in an hour. We have a customer-facing website, but there's very little there that can't be served by a cache.

tbh, I can't recall a support request from a customer that was caused by our infrastructure vendor and not our product.

Post reply on HN