111–120 of 410 posts
My favorite sentence: "Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event."
The complexity that AWS has to deal with is astounding. Sure having your main production network and a management network is common. But making sure all of it scales and doesn't bring down the other is what I think they are dealing with here. It must have been crazy hard to troubleshoot when you are flying blind because all your monitoring is unresponsive. Clearly more isolation with clearly delineated information ex…
“But AWS has more operations staff than I would ever hope to hire” — a common mantra when talking about using the cloud overall. I’m not saying I fully disagree. But consolidation of the worlds hosting necessitates a very complicated platform and these things will happen, either due to that complexity, failures that can’t be foreseen or good old fashioned Sod’s law. I know AWS marketing wants you to believe it’s all…
My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…
I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…
Jeez, how many meetings did you go to, and how long was this person's vacation? I'm jelly of being allowed to take that much time off continuously.
Earlier quoted context omitted.
Multiple AWS employees have acknowledged it takes VP approval to change the status color of the dashboard. That is absurd and it tells you everything you need to know. The status page isn't about accurate information, it's about plausible deniability and keeping AWS out of the news cycle.
>it's about plausible deniability and keeping AWS out of the news cycle. How'd that work out for them? https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=w...
My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…
if ($pain > $gain) {
move_your_shit_and_exit_aws();
}
sub move_your_shit_and_exit_aws
{
printf("Dude. We have too much pain. Start moving\n");
printf("Yeah. That won't happen, so who cares\n");
exit(1);
}Having an internal network like this that everything on the main AWS network so heavily depends on is just bad design. One does not create a stable high tech spacecraft and then fuels it with coal.
Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.
Our SOP is to cut over to a second region the moment we see any AZ-level shenanigans. We've been burned too often.
> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event. What does this even mean ? I bet most people use DynamoDB via a VPC, in a Lambda or in EC2
Your application can call DynamoDB via the public endpoint (dynamodb.us-east-1.amazonaws.com). But if you're in a VPC (i.e. practically all AWS workloads in 2021), you have to route to the internet (you need public subnet(s) I think) to make that call. VPC Endpoints create a DynamoDB endpoint in your VPC, from the documentation: "When you create a VPC endpoint for DynamoDB, any requests to a DynamoDB endpoint within…
Earlier quoted context omitted.
I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…
The problem here sounds like lack of clarity over the meaning of the colours. In organisations with 100s of in-flight projects, it’s understandable that red is reserved for projects that are causing extremely serious issues right now. Otherwise, so many projects would be red, that you’d need a new colour.