I just noticed Stripe's API returning intermittent 500 errors for the first time in forever. Now I know why! =)
yeah so did we.. and then also from auth0. aws status page was next thing to check.
AWS West Outage (US-WEST-2 region)
11–20 of 24 posts
Re: AWS West Outage (US-WEST-2 region)
#12These things happen to every datacentre. That’s why AWS has multiple, separately managed regions that you can use to keep your app up with.
and pay double? no way
It's not paying double. Take the total number of instances serving the app and divide them across data centers and availability zones within datacenters. This might require adding a few of servers if the app is replicated across a very small number of instances, but for apps that already have a significant number of instances to handle the capacity, it's just a matter of rearranging things.
Depending on how quickly new instances of the app can be brought up to, it may be worthwhile to run a bit more capacity so that the remaining instances don't melt down should all the instances in one zone become unavailable.
Re: AWS West Outage (US-WEST-2 region)
#13These things happen to every datacentre. That’s why AWS has multiple, separately managed regions that you can use to keep your app up with.
That's why my aws console wasn't working properly this afternoon. It seems that Amazon is not using it's services correctly
Re: AWS West Outage (US-WEST-2 region)
#14Seems to have hit pagerduty pretty hard... your poor on-call guy may have no idea his app is busted.
Interesting. I got a page during the incident, so at least some of it was working.
Re: AWS West Outage (US-WEST-2 region)
#15These things happen to every datacentre. That’s why AWS has multiple, separately managed regions that you can use to keep your app up with.
But AWS has the concept of availability zones that they equate to a data center. AWS refers to an AZ as an "isolated location". It's becoming clear with the S3 outages and this outage that isolation is not as robust as expected so now we see more reference to cross-region replication. But unlike cross-AZ services like S3 or DynamoDB, you need to build your own cross-region replication and failover.
Re: AWS West Outage (US-WEST-2 region)
#16These things happen to every datacentre. That’s why AWS has multiple, separately managed regions that you can use to keep your app up with.
But AWS has the concept of availability zones that they equate to a data center. AWS refers to an AZ as an "isolated location". It's becoming clear with the S3 outages and this outage that isolation is not as robust as expected so now we see more reference to cross-region replication. But unlike cross-AZ services like S3 or DynamoDB, you need to build your own cross-region replication and failover.
Yep. My take: humans fail at more than hardware does, and AWS takes a lot of action per region.
Re: AWS West Outage (US-WEST-2 region)
#17Yeah noticed this in our production app today. Thanks AWS. Thank goodness we had multi AZ DB running and other failovers.
Re: AWS West Outage (US-WEST-2 region)
#18Would they be publishing a post-mortem? If yes, where do I get to read it? Someone be kind enough to enlighten me. Sincerely,
Re: AWS West Outage (US-WEST-2 region)
#19Seems to have hit pagerduty pretty hard... your poor on-call guy may have no idea his app is busted.
Pager as in - the electronic device that existed before the cellphone?