Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

621–630 of 1001 posts

Re: AWS multiple services outage in us-east-1

#621

Post by 1970-01-01 »

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

I remember Facebook had a similar story when they botched their BGP update and couldn't even access the vault. If you have circular auth, you don't have anything when somebody breaks DNS.

Re: AWS multiple services outage in us-east-1

#622

Post by samcat116 »

masfuerte wrote: post #592
JCM9 wrote: post #487

US-East-1 is more than just a normal region. It also provides the backbone for other services, including those in other regions. Thus simply being in another region doesn’t protect you from the consistent us-east-1 shenanigans. AWS doesn’t talk about that much publicly, but if you press them they will admit in private that there are some pretty nasty single points of failure in the design of AWS that can materialize…

Amazon are planning to launch the EU Sovereign Cloud by the end of the year. They claim it will be completely independent. It may be possible then to have genuine resiliency on AWS. We'll see.

This is the difference between “partitions” and “regions”. Partitions have fully separate IAM, DNS names, etc. This is how there are things like US Gov Cloud, the Chinese AWS cloud, and now the EU sovereign cloud

Re: AWS multiple services outage in us-east-1

#624

Post by oofbey »

wwdmaxwell wrote: post #458
chibea wrote: post #443

One main problem that we observed was that big parts of their IAM / auth setup was overloaded / down which led to all kinds of cascading problems. It sounds as if Dynamo was reported to be a root cause, so is IAM dependent on dynamo internally? Of course, such a large control plane system has all kinds of complex dependency chains. Auth/IAM seems like such a potentially (global) SPOF that you'd like to reduce depende…

I think Amazon uses an internal platform called Dynamo as a KV store, it’s different than DynamoDB, so im thinking the outage could be either a dns routing issue or some kind of node deployment problem. Both of which seem to prop up in post mortems for these widespread outages.

They said the root cause was DNS for dynamoDB. inside AWS relying on dynamoDB is highly encouraged so it’s not surprising that a failure there would cascade broadly. The fact that EC2 instance launching is effected is surprising. Loops in the service dependency graph are known to be a bad idea.

Re: AWS multiple services outage in us-east-1

#625

Post by indoordin0saur »

padjo wrote: post #369
jacquesm wrote: post #353

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

Been doing this for about 8 years and I've worked through a serious AWS disruption at least 5 times in that time.

Re: AWS multiple services outage in us-east-1

#626

Post by archon810 »

tonypapousek wrote: post #419

Looks like they’re nearly done fixing it. > Oct 20 3:35 AM PDT > The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2…

Except it just broke again.

Re: AWS multiple services outage in us-east-1

#627

Post by archon810 »

kuon wrote: post #466

I realize that my basement servers have better uptime than AWS this year! I think most sysadmin don't plan for AWS outage. And economically it makes sense. But it makes me wonder, is sysadmin a lost art?

That's not very surprising. At this point you could say that your microwave has a better uptime. The complexity comparison to all the Amazon cloud services and infrastructure would be roughly the same.

Re: AWS multiple services outage in us-east-1

#628

Post by avs733 »

ibejoeb wrote: post #577

This is just a silly anecdote, but every time a cloud provider blips, I'm reminded. The worst architecture I've ever encountered was a system that was distributed across AWS, Azure, and GCP. Whenever any one of them had a problem, the system went down. It also cost 3x more than it should.

i'll bet there are a large number of systems that are dependent on multiple cloud platforms being up without even knowing it. They run on AWS, but rely on a tool from someone else that runs on GCP or on Azure and they haven't tested what happens if that tools goes down...

Common Cause Failures and false redundancy are just all over the place.

Re: AWS multiple services outage in us-east-1

#629

Post by shkkmo »

tonypapousek wrote: post #419

Looks like they’re nearly done fixing it. > Oct 20 3:35 AM PDT > The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2…

Still not fixed and may have gotten worse...
Post reply on HN