Earlier quoted context omitted.
TIL Amazon owns IMDB
Yeah, I was also surprised when I learned this. Another surprising thing is that they own it since 1998.
And it was all in Perl up until a few years ago (rewritten in Java).
AWS us-east-1 outage
931–940 of 1001 posts
Re: AWS us-east-1 outage
#932Earlier quoted context omitted.
I can't remember seeing problems be more strongly worded than "Increased Error Rates" or "high error rates with S3 in us-east-1" during the infamous S3 outage of 2017 - and that was after they struggled to even update their own status page because of S3 being down. :)
During the Facebook outage FB wrote something along the lines of "We noticed that some users are experiencing issues with our apps" eventhough nothing worked anymore
Their entire infrastructure was unroutable for a while if I remember correctly. That is a euphemism if I ever saw one.
Re: AWS us-east-1 outage
#933Earlier quoted context omitted.
> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…
Every damn Well-Architected Framework includes multi-AZ if not multi-region redundancy, and yet the single access point for their millions of customers is single-region. Facepalm in the form of $100Ms in service credits.
Don’t think there is an sla for the console , so you would not be claiming anything for the console at least
Re: AWS us-east-1 outage
#934Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…
They are still lying about it, the issues are not only affecting the console but also AWS operations such as S3 puts. S3 still shows green.
I still can't create/destroy/etc CloudFront distros. They are stuck in "pending" indefinitely.
Re: AWS us-east-1 outage
#935Earlier quoted context omitted.
SLAs. Officially acknowledging an incident means that they now have to issue the SLA credits.
The outage dashboard is normally only updated if a certain $X percent of hosts / service is down. If the EC2 section were updated every time a rack in a datacenter went down, it would be red 24x7. It's only updated when a large percentage of customers are impacted, and most of the time this number is less than what the HN echo chamber makes it appear to be.
A rack out isn’t going to cause as much an issue as we’re seeing. Most people wouldn’t even notice.
This is a situation that’s fairly significant. Degraded isn’t really what’s happening considering how much is borked — including their own console. That’s an outage. That’s not an echo chamber situation when basically an entire DC is inaccessible
Re: AWS us-east-1 outage
#936Earlier quoted context omitted.
Can’t comment on most of your post but I know a lot of Amazon engineers who think of the CoE process (Correction of Error, what other companies would call a postmortem) as punitive
They aren't meant to be, but shitty teams are shitty. You can also create a COE and assign it to another team. When I was at AWS, I had a few COEs assigned to me by disgruntled teams just trying to make me suffer and I told them to pound sand. For my own team, I wrote COEs quite often and found it to be a really great process for surfacing systemic issues with our management chain and making real improvements, but it…
At some point the number of people who were on shitty teams becomes an indictment on the wider culture at Amazon.
Re: AWS us-east-1 outage
#937Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…
> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…
> At a different (unnamed) FAANG
I'm guessing Google, on the basis of the recently published (to the public) "I just want to serve 5TB"[1] video. If it isn't Google, then the broccoli man video is still a cogent reminder that unyielding multi-region rigor comes with costs.
Re: AWS us-east-1 outage
#938Earlier quoted context omitted.
Suggesting that when the status page sends a status request and hears no response—it defaults to green—hear no evil and see no evil —> report no evil Either way—overt lies or engineering incompetence—it’s disappointing!
Pretty low chance that the status page is automated, especially via health checks. I imagine it's a static asset updated by hand.
Or the service that updates the status page runs out of us-east-1.
Re: AWS us-east-1 outage
#939Earlier quoted context omitted.
> Yes, VP approval is needed to make any updates on the status dashboard. But that's not as hard as it may seem. AWS executives are extremely operation-obsessed, and when there is an outage of any size are engaged with their service teams immediately. My experience generally aligns with amzn-throw, but this right here is why. There's a manual step here and there's always drama surrounding it. The process to update th…
It's intentional plausible deniability. By creating the manual step you can shift blame away. It's just like the concept of personal health dashboards which are designed to keep an asymmetry in reliability information from a host and the client to their personal anecdata experiences. Ontop of all of this, the metrics are pretty arbitrary. Let's not pretend businesses haven't been intentionally advertising in deceitfu…
Here's one business honest with their status page: https://news.ycombinator.com/item?id=29478679
Re: AWS us-east-1 outage
#940Earlier quoted context omitted.
Would they? Having 3 outages in a year sounds like an organization problem. Not enough safeguards to prevent very routine human errors. But instead of worrying about that we just assign a guy to take the fall
If you work in a technical role and you _don't_ have the ability to break something, you're unlikely to be contributing in a significant way. Likely that would make you a junior developer whose every line of code is heavily scrutinized. Engineers should be experts and you should be able to trust them to make reasonable choices about the management of their projects. That doesn't mean there can't be some checks in pla…
The tactical point is to remove sharp edges, eg there's a tool that optionally take a region argument.
network_cli remove_routes [--region us-east-1]
network_cli remove_routes
All of the tools need to not default to breaking the world. That is the first and foremost thing being pushed. If an engineer is remotely afraid to come forwards (beyond self-shame/judgement) after an incident, and say "hey, I accidentally did this thing", then the situation will never get any better.
That doesn't mean that engineers don't have the ability to break things, but it means it's harder (and very intentionally so) for a stressed out human operator to do the wrong thing by accident. Accidents happen. Do you just plan on never getting into a car accident, or do you wear a seat belt?