Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#21

Post by dijit »

amalter wrote: post #19
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Once is a mistake.

Twice is a coincidence.

Three times is a pattern.

But this… This is every time.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#22

Post by nayuki »

> Operators instead relied on logs to understand what was happening and initially identified elevated internal DNS errors. Because internal DNS is foundational for all services and this traffic was believed to be contributing to the congestion, the teams focused on moving the internal DNS traffic away from the congested network paths. At 9:28 AM PST, the team completed this work and DNS resolution errors fully recovered.

Having DNS problems sounds a lot like the Facebook outage of 2021-10-04. https://en.wikipedia.org/wiki/2021_Facebook_outage

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#23

Post by [deleted] »

foobarbecue wrote: post #17

"... the networking congestion impaired our Service Health Dashboard tooling from appropriately failing over to our standby region. By 8:22 AM PST, we were successfully updating the Service Health Dashboard." Sounds like they lost the ability to update the dashboard. HN comments at the time were theorizing it wasn't being updated due to bad policies (need CEO approval) etc. Didn't even occur to me that it might be st…

[deleted]

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#24

Post by Clubber »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

Yes, it's a conflict of interest. They have a guarantee on uptime and they decide what their actual uptime is. There's a lot of that now. Most insurances comes to mind.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#25

Post by bamboozled »

Still doesn’t explain the cause of all the IAM permission denied requests we saw against policies which are again working fine without any intervention.

Obviously networking issues can cause any number of symptoms but it seems like an unusual detail to leave out to me. Unless it was another ongoing outage happening at the same time.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#26

Post by jtoberon »

simlevesque wrote: post #16

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event. What does this even mean ? I bet most people use DynamoDB via a VPC, in a Lambda or in EC2

VPC Endpoint is a feature of VPC: https://docs.aws.amazon.com/vpc/latest/privatelink/endpoint-...

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#27

Post by propter_hoc »

dijit wrote: post #13
propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

I agree with you, but my services are actually in Canada (Central). There's only one region in Canada, so I don't really have an alternative. AWS justifies it by saying there are three AZs (distinct data centres) within Canada (Central), but I get scared when I see these region-wide issues. If the AZs were really distinct, you wouldn't really have region-wide issues.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#28

Post by [deleted] »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

[deleted]

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#30

Post by discodave »

simlevesque wrote: post #16

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event. What does this even mean ? I bet most people use DynamoDB via a VPC, in a Lambda or in EC2

Your application can call DynamoDB via the public endpoint (dynamodb.us-east-1.amazonaws.com). But if you're in a VPC (i.e. practically all AWS workloads in 2021), you have to route to the internet (you need public subnet(s) I think) to make that call.

VPC Endpoints create a DynamoDB endpoint in your VPC, from the documentation:

"When you create a VPC endpoint for DynamoDB, any requests to a DynamoDB endpoint within the Region (for example, dynamodb.us-west-2.amazonaws.com) are routed to a private DynamoDB endpoint within the Amazon network. You don't need to modify your applications running on EC2 instances in your VPC. The endpoint name remains the same, but the route to DynamoDB stays entirely within the Amazon network, and does not access the public internet."

Post reply on HN