Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

191–200 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#191
post #89

Having an internal network like this that everything on the main AWS network so heavily depends on is just bad design. One does not create a stable high tech spacecraft and then fuels it with coal.

It's 2006, you work for an 'online book store' that's experimenting with this cloud thing. Are you going to build a whole new network involving multi-million dollar networking appliances?

Developing EC2 did involve building a whole new network. It was a new service, built from the ground up to be a public product.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#192

Earlier quoted context omitted.

I've been naively setting up our distributed databases in separate AZs for a couple years now, paying, sometimes, thousands of dollars per month in data replication bandwidth egress fees. As far as I can remember I've never never seen an AZ go down, and the only region that has gone down has been us-east-1.

> I've never never seen an AZ go down, and the only region that has gone down has been us-east-1. Doesn't the region going down mean that _all_ its AZs have gone down? Or is my mental model of this incorrect?

No. See https://aws.amazon.com/about-aws/global-infrastructure/regio...

A region is a networking paradigm. An AZ is a group of 2-6 data centers in the same city more or less.

If a region goes down or is otherwise impacted, its AZs are unavailable or similar.

If an AZ goes down, your VMs in said centers are disrupted in the most direct sense.

It's the difference between loss of service and actual data loss.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#193

> The AWS container services, including Fargate, ECS and EKS, experienced increased API error rates and latencies during the event. While existing container instances (tasks or pods) continued to operate normally during the event, if a container instance was terminated or experienced a failure, it could not be restarted because of the impact to the EC2 control plane APIs described above. This seems pretty obviously f…

Strong agree. We were using Fargate nodes in our us-east-1 EKS cluster and not all of our nodes dropped, but every coredns pod did. When they came back up their age was hours older than expected, so maybe a problem between Fargate and the scheduler rendered them “up” but unable to be reached? Either way, was surprising to us that already provisioned compute was impacted.

Saw the same. The only cluster services I was running in Fargate were CoreDNS and cluster-autoscaler; thought it would help the clusters recover from anything happening to the node group where other core services run. Whoops.

Couldn't just delete the Fargate profile without a working EKS control plane. I lucked out in that the label selector the kube-dns Service used was disjoint from the one I'd set in the Fargate profile, so I just made a new "coredns-emergency" deployment and cluster networking came back. (cluster-autoscaler was moot since we couldn't launch instances anyway.)

I was hoping to see something about that in this announcement, since the loss of live pods is nasty. Not inclined to rely on Fargate going forward. It is curious that you saw those pod ages; maybe Fargate kubelets communicate with EKS over the AWS internal network?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#194
post #156

Earlier quoted context omitted.

At the large platform company where I work, our policy is if the customer reported the issue before our internal monitoring caught it, we have failed. Give 5 minutes for alerting lag, 10 minutes to evaluate the magnitude of impact, 10 minutes to craft the content and get it approved, 5 minutes to execute the update, adds up to 30 minutes end to end with healthy buffer at each step. 1 hour (52 minutes according to the…

> our policy is if the customer reported the issue before our internal monitoring caught it They've discovered it right away, the Service Health Dashboard was not updated. source: link.

They don’t say explicitly right away do they? I skimmed twice.

But yes you’re right, there’s no reason to question their monitoring or alerting specifically.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#195
post #114

Earlier quoted context omitted.

>it's about plausible deniability and keeping AWS out of the news cycle. How'd that work out for them? https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=w...

When is the last time they had a single service outage in a single region? How about in a single AZ in a single region? Struggling to find a lot of headline stories? I'm willing to bet it's happened in the last 2 years and yet I don't see many news articles about it... so I'd say if the only thing that hits the front page is a complete region outage for 6+ hours, it's working out pretty well for them.

Last year's Thanksgiving outage and this one are the two biggest. They've been pretty reliable. That's still 99.7% uptime.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#197

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

Exactly, we had the same thing almost exactly a year ago - https://www.dailymail.co.uk/sciencetech/article-8994907/Wide...

They are barely doing better than 2 9s.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#199
post #173
post #32

Earlier quoted context omitted.

>You can take it on bad faith It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless. If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.

>>It's smart politics -- I don't blame them Um, so you think straight-up lying is good politics? Any 7-year old knows that telling a lie when you broke something makes you look better superficially, especially if you get away with it. That does not mean that we should think it is a good idea to tell lies when you break things. It sure as hell isn't smart politics in my book. It is straight-up disqualifying to do busi…

You don’t know what you’re talking about.

AWS spends a lot of time thinking about this problem in service to their customers.

How do you reduce the status of millions of machines, the software they run, and the interconnected-ness of those systems to a single graphical indicator?

It would be dumb and useless to turn something red every single time anything had a problem. Literally there are hundreds of things broken every minute of every day. On-call engineers are working around the clock on these problems. Most of the problems either don’t affect anyone due to redundancy or affect only a tiny number of customers- a failed memory module or top-of-rack switch or a random bit flip in one host for one service.

Would it help anyone to tell everyone about all these problems? People would quickly learn to ignore it as it had no bearing on their experience.

What you’re really arguing is that you don’t like the thresholds they’ve chosen. That’s fine, everyone has an opinion. The purpose of health dashboards like these are mostly so that customers can quickly get an answer to “is it them or me” when there’s a problem.

As others on this thread have pointed out, AWS has done a pretty good job of making the SHD align with the subjective experience of most customers. They also have personal health dashboards unique to each customer, but I assume thresholding is still involved.

Post reply on HN