Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#81

Post by electroly »

logical_proof wrote: post #56
electroly wrote: post #43

Earlier quoted context omitted.

> The entire time their outage board was solid green Unless you're talking about some board other than the Service Health Dashboard, this isn't true. They dropped EC2 down to degraded pretty early on. I bemusedly noted in our corporate Slack that every time I refreshed the SHD, another service was listed as degraded. Then they added the giant banner at the top. Their slight delay in updating the SHD at the beginning…

That is not true. There was hours before they started annotating any kind of service issues. Maybe from when you noticed there was a problem it appeared to be quick, but the board remained green for a large portion of the outtage.

No, it was about an hour. We were aware from the very moment EC2 API error rates began to elevate, around 10:30 Eastern. By 11:30 the dashboard was updating. This timing is mentioned in the article, and it all happened in the middle of our workday on the east coast. The outage then continued for about 7 hours with SHD updates. I suspect we actually both agree on how long it took them to start updating, but I conclude that 1 hour wasn't so bad.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#82

Post by anonu »

codegeek wrote: post #79
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

"Our Support Contact Center also relies on the internal AWS network, so the ability to create support cases was impacted from 7:33 AM until 2:25 PM PST. " This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here. Also, i…

Wasn't this the Bezos directive early on that created AWS? Anything that was created had to be a service with an API. Not allowed to recreate the wheel. So AWS depends on AWS.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#83

Post by banana_giraffe »

propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

It can be a bit hard to know, since the AZ identifiers are randomized per account, so if you think you have problems in us-west-1a, I can't check on my side. You can get the AZ ID out of your account to de-randomize things, so we can compare notes, but people rarely bother, for whatever reason.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#84

Post by jiggawatts »

systemvoltage wrote: post #54
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

People of HN has been extremely unprofessional with regards to AWS's downtime. Some kind of a massive zeitgeist against Amazon, like a giant hive mind that spews hate. Why are we doing this folks? What's making you so angry and contemptful? Literally try searching the history of downtimes and it was always professional and respectful. Yesterday, my comment was fricking flagged for asking people to be nice to which pe…

AWS as a business has an enormous (multi-billion-dollar) moral hazard: they have a fantastically strong disincentive to update their status dashboard to accurately reflect the true nature of an ongoing outage. They use weasel words like "some customers may be seeing elevated errors", which we all know translates to "almost all customers are seeing 99.99% failure rates."

They have a strong incentive to lie, and they're doing it. This makes people dependent upon the truth for refunds understandably angry.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#85

Post by JCM9 »

tristor wrote: post #74
JCM9 wrote: post #55

Queue the armchair infrastructure engineers. The reality is that there’s a handful of people in the world that can operate systems at this sheer scale and complexity and I have mad respect for those in that camp.

Some of us are in that camp and are looking at this outage and also pointing out that they continuously fail to accurately update their status dashboard in this and prior outages. Yes, doing what AWS does is hard, and yes outages /will/ happen, it is no knock on them that this outage occurred, what is a knock is that they haven't communicated honestly while the outage was ongoing.

They address that in the post, and between Twitter, HN and other places there wasn’t anyone legit questioning if something was actually broken. Contacts at AWS also all were very clear that yes something was going on and being investigated. This narrative that AWS was pretending nothing was wrong just wasn’t true based on what we saw.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#86

Post by walrus01 »

codegeek wrote: post #79
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

"Our Support Contact Center also relies on the internal AWS network, so the ability to create support cases was impacted from 7:33 AM until 2:25 PM PST. " This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here. Also, i…

I know a few tiny ISPs that host their voip server and email server outside of their own ASN so that in the event of a catastrophic network event, communications with customers is still possible... Not saying amazon should do the same, but the general principle isn't rocket science.

there's such as thing as too much dogfooding.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#87

Post by banana_giraffe »

DenisM wrote: post #49

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.

Yeah

> For example, while running EC2 instances were unaffected by this event

That ignores the 3-5% drop in traffic I saw in us-east-1 on EC2 instances that only talk to peers on the Internet with TCP/IP during this event.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#88

Post by almostdeadguy »

> The AWS container services, including Fargate, ECS and EKS, experienced increased API error rates and latencies during the event. While existing container instances (tasks or pods) continued to operate normally during the event, if a container instance was terminated or experienced a failure, it could not be restarted because of the impact to the EC2 control plane APIs described above.

This seems pretty obviously false to me. My company has several EKS clusters in us-east-1 with most of our workloads running on Fargate. All of our Fargate pods were killed and were unable to be restarted during this event.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#90

Post by meabed »

ALittleLight wrote: post #48
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

Working with consultants for years and that’s exactly the playbook, project might not even launch after 2 years of deadlines while status is yellow or green and never red.
Post reply on HN