Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#31

Post by shepherdjerred »

nayuki wrote: post #22

> Operators instead relied on logs to understand what was happening and initially identified elevated internal DNS errors. Because internal DNS is foundational for all services and this traffic was believed to be contributing to the congestion, the teams focused on moving the internal DNS traffic away from the congested network paths. At 9:28 AM PST, the team completed this work and DNS resolution errors fully recove…

It’s quite a bit different… Facebook took themselves offline completely because of a bad BGP update, whereas AWS had network congestion due to a scaling event. DNS relies on the network, so of course it’ll be impacting if networking is also impacted.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#32

Post by s_dev »

amalter wrote: post #19
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

>You can take it on bad faith

It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless.

If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#33

Post by human »

nayuki wrote: post #22

> Operators instead relied on logs to understand what was happening and initially identified elevated internal DNS errors. Because internal DNS is foundational for all services and this traffic was believed to be contributing to the congestion, the teams focused on moving the internal DNS traffic away from the congested network paths. At 9:28 AM PST, the team completed this work and DNS resolution errors fully recove…

The rule is that it’s always DNS.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#35

Post by EsotericAlgo »

llaolleh wrote: post #18

I wonder if they could've designed better circuit breakers for situations like this. They're very common in electrical engineering, but I don't think they're as common in software design. Something we should try to design and put in, actually for situations like this.

They’re a fairly common design pattern https://en.m.wikipedia.org/wiki/Circuit_breaker_design_patte.... However, they certainly aren’t implemented with the frequency they should be at service level boundaries resulting in these sorts of cascading failures.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#36

Post by randmeerkat »

dijit wrote: post #13
propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

AWS has been getting a pass on their stability issues in us-east-1 for years now because it’s their “oldest” zone. Maybe they should invest in fixing it instead of inventing new services to sell.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#37

Post by riknos314 »

llaolleh wrote: post #18

I wonder if they could've designed better circuit breakers for situations like this. They're very common in electrical engineering, but I don't think they're as common in software design. Something we should try to design and put in, actually for situations like this.

One of the big issues mentioned was that one of the circuit breakers they did have (client back off), didn't function properly. So they did have a circuit breaker in the design, but it was broken.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#38

Post by wongarsu »

foobarbecue wrote: post #17

"... the networking congestion impaired our Service Health Dashboard tooling from appropriately failing over to our standby region. By 8:22 AM PST, we were successfully updating the Service Health Dashboard." Sounds like they lost the ability to update the dashboard. HN comments at the time were theorizing it wasn't being updated due to bad policies (need CEO approval) etc. Didn't even occur to me that it might be st…

Hasn't this exact thing (something in US-east-1 goes down, AWS loses ability to update dashboard) happened before? I vaguely remember it was one of the S3 outages, but I might be wrong.

In any case, AWS not updating their dashboard is almost a meme by now. Even for global service outages the best you will get is a yellow.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#39

Post by all2well »

simlevesque wrote: post #16

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event. What does this even mean ? I bet most people use DynamoDB via a VPC, in a Lambda or in EC2

From within a VPC, you can either access DynamoDB via its public internet endpoints (eg, dynamodb.us-east-1.amazonaws.com, which routes through an Internet Gateway attachment in your VPC), or via a VPC endpoint for dynamodb that's directly attached to your VPC. The latter is useful in cases where you want a VPC to not be connected to the internet at all, for example.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#40

Post by notimetorelax »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

If you were deployed in 2 regions would it alleviate the impact?
Post reply on HN