Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

901–910 of 1001 posts

Re: AWS us-east-1 outage

#901

Post by markus_zhang »

JPKab wrote: post #862
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

As a user of Sagemaker in us-east-1, I deeply fucking resent AWS claiming the service is normal. I have extremely sensitive data, so Sagemaker notebooks and certain studio tools make sense for me. Or DID. After this I'm going back to my previous formula of EC2 and hosting my own GPU boxes. Sagemaker is not working, I can't get to my work (notebook instance is frozen upon launch, with zero way to stop it or restart it…

Looks like they removed some 9s from availability in one day. I wonder if more are considering moving away from cloud.

Re: AWS us-east-1 outage

#902

Post by abdabab »

wolverine876 wrote: post #852
abdabab wrote: post #846

Earlier quoted context omitted.

If an engineer causes an outage by mistake and then ensures that would never happen again, he has made a positive impact.

I understand that, but eventually they need to evaluate performance, for promotions, demotions, raises, cuts, hiring, firing, etc. How is that done?

It’s standard. Career ladder [1] sets expectation for each level. Performance is measured against those expectations. Outages don’t negatively impact a single engineer.

The key difference is the perspective. If reliability is bad that’s an organizational problem and blaming or punishing one engineer won’t fix that.

[1] An example ladder from Patreon: https://levels.patreon.com/

Re: AWS us-east-1 outage

#903

Post by sneak »

vdqtp3 wrote: post #872
sneak wrote: post #734

Earlier quoted context omitted.

Now think of how many assets of various governments' militaries are discreetly employed as normal operational staff by FAAMG in the USA and have access to cause such events from scratch. I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies to this end, because they are invested in that Not Being Possible (indeed, it's their job). There is a societal resilience benefit t…

> I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies This comment is how I know you don't work in the public sector. Those agencies' infrastructures are essentially run by contractors with a few GS personnel making bad decisions every chance they get and a few DoD personnel acting like their rank can fix technical problems.

I'm not talking about running infrastructure, I'm talking about working with HR and making sure that the people they've hired as sysadmins aren't meeting with FSB or PLA agents in a local park on weekends and accepting suitcases of USD cash to accidentally `rm -rf` all of the Zookeeper/etcd nodes at once on a Monday morning.

Re: AWS us-east-1 outage

#904

Post by twistedpair »

s_dev wrote: post #666
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

There is a company that delivers broadcast video ads to hundreds of TV stations on demand. The ad has to run and run now, so they cannot tolerate failure.

They write the videos to GCS storage in Google Cloud, and to S3 in AWS. Every point of their workflows are checkpointed and cross referenced across GCP and AWS. If either side drops the ball, the other picks it up.

So yes, you can design a super fault tolerant system. This company did it because failing to deliver a few ads would mean lose of major contracts.

Re: AWS us-east-1 outage

#906

Post by notwedtm »

codegeek wrote: post #724
amzn-throw wrote: post #444

Earlier quoted context omitted.

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

"Yes, VP approval is needed to make any updates on the status dashboard." If services are clearly down, why is this needed ? I can understand the oversights required for a company like Amazon but this sounds strange to me. If services are clearly down, I want that damn status update right away as a customer.

Because "services down" also means SLA credits.

Re: AWS us-east-1 outage

#907

Post by balls187 »

jesboat wrote: post #484
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

MAANG*

How long before Meta takes over for Facebook?

Re: AWS us-east-1 outage

#908

Post by bamboozled »

Rantenki wrote: post #739
jeremyjh wrote: post #271

Earlier quoted context omitted.

They are still lying about it, the issues are not only affecting the console but also AWS operations such as S3 puts. S3 still shows green.

Yep, I am seeing failures on IAM as well: aws iam list-policies An error occurred (503) when calling the ListPolicies operation (reached max retries: 2): Service Unavailable

I'm seeing errors for things that worked fine, like policies that had no issue now are saying "access denied".

I'm wondering if the cause of the outage has to do with something changing in the way IAM is interpreted ?

Re: AWS us-east-1 outage

#909

Post by rampole »

amzn-throw wrote: post #444
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

I'm an ex-Amazon employee and approve of this response.

It reflects exactly my experience there.

Blameless post-mortem, stick to the facts and how the situation could be avoided/reduced/shortened/handled better for next time.

In fact, one of the guidelines for writing COE (Correction Of Error, Amazon's jargon for Post Mortem) is that you never mention names but use functions and if necessary teams involved:

1. Personal names don't mean anything except to the people who were there on the incident at the time. Someone reading the CoE on the other side of the world or 6 months from now won't understand who did what and why. 2. It stands in the way of honest accountability.

Re: AWS us-east-1 outage

#910

Post by sitharus »

vkgfx wrote: post #839
all_usernames wrote: post #560

Earlier quoted context omitted.

Every damn Well-Architected Framework includes multi-AZ if not multi-region redundancy, and yet the single access point for their millions of customers is single-region. Facepalm in the form of $100Ms in service credits.

>Facepalm in the form of $100Ms in service credits. Part of me wonders how much they're actually going to pay out, given that their own status page has only indicated five services with moderate ("Increased API Error Rates") disruptions in service.

That public status page has no bearing on service credits, it's a statically hosted page updated when there's significant public impact. A lot of issues never make it there.

Every AWS customer has a personal health dashboard that links the issues to their services which is updated much faster, and links issues to your affected resources. Additionally requests for credits are done by the customer service team who have even more information.

Post reply on HN