Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#11

Post by [deleted] »

propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

[deleted]

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#13

Post by dijit »

propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone.

So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#16

Post by simlevesque »

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event.

What does this even mean ? I bet most people use DynamoDB via a VPC, in a Lambda or in EC2

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#17

Post by foobarbecue »

"... the networking congestion impaired our Service Health Dashboard tooling from appropriately failing over to our standby region. By 8:22 AM PST, we were successfully updating the Service Health Dashboard."

Sounds like they lost the ability to update the dashboard. HN comments at the time were theorizing it wasn't being updated due to bad policies (need CEO approval) etc. Didn't even occur to me that it might be stuck in green mode.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#18

Post by llaolleh »

I wonder if they could've designed better circuit breakers for situations like this. They're very common in electrical engineering, but I don't think they're as common in software design. Something we should try to design and put in, actually for situations like this.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#19

Post by amalter »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.
Post reply on HN