Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
1–10 of 410 posts
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#2Exceeded character limit on the title so I couldn't include this detail there, but this is the post-mortem of the event on December 7 2021.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#3Between this and Log4j, I'm just glad it's Friday.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#4Obviously one hopes these things don’t happen, but that’s an impressive and transparent write up that came out quickly.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#5The complexity that AWS has to deal with is astounding. Sure having your main production network and a management network is common. But making sure all of it scales and doesn't bring down the other is what I think they are dealing with here.
It must have been crazy hard to troubleshoot when you are flying blind because all your monitoring is unresponsive. Clearly more isolation with clearly delineated information exchange points are needed.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#6I wish it contained actual detail and wasn’t couched in generalities.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#7Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#8The complexity that AWS has to deal with is astounding. Sure having your main production network and a management network is common. But making sure all of it scales and doesn't bring down the other is what I think they are dealing with here. It must have been crazy hard to troubleshoot when you are flying blind because all your monitoring is unresponsive. Clearly more isolation with clearly delineated information ex…
“But AWS has more operations staff than I would ever hope to hire” — a common mantra when talking about using the cloud overall.
I’m not saying I fully disagree. But consolidation of the worlds hosting necessitates a very complicated platform and these things will happen, either due to that complexity, failures that can’t be foreseen or good old fashioned Sod’s law.
I know AWS marketing wants you to believe it’s all magic and rainbows, but it’s still computers.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#9Yeah, cloudwatch APIs went to the drain. Good for them for publishing this at least.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#10My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone.
Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by their "green board" is meaningless. During the event they were late and reluctant to report it and its significance. My point is that they are wrongly incentivized to keep the board green at all costs.