Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

161–170 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#161

Post by koheripbal »

luhn wrote: post #67
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this. They "own up to it" in their postmortems,…

This challenge is not specific to Amazon.

Being able to automatically detect system health is a non-trivial effort.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#162

Post by tootie »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

Second hand info but supposedly when an outage hits they go all hands on resolving it and no one who knows what's going on has time to update the status board which is why it's always behind.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#163

Post by bob778 »

koheripbal wrote: post #161
luhn wrote: post #67

Earlier quoted context omitted.

Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this. They "own up to it" in their postmortems,…

This challenge is not specific to Amazon. Being able to automatically detect system health is a non-trivial effort.

That’s not what’s being asked though - in all 3 events, they couldn’t manually update it. It’s clearly not a priority to fix it for even manual alerts.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#164

Post by blackearl »

koheripbal wrote: post #161
luhn wrote: post #67

Earlier quoted context omitted.

Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this. They "own up to it" in their postmortems,…

This challenge is not specific to Amazon. Being able to automatically detect system health is a non-trivial effort.

Why automatic? Surely someone could have the responsibility to do it manually.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#165

Post by tootie »

discodave wrote: post #103
cyounkins wrote: post #42

My favorite sentence: "Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event."

I saw pleeeeeenty of untested code at Amazon/AWS. Looking back it was almost like the most important services/code had the least amount of testing. While internal boondoggle projects (I worked on a couple) had complicated test plans and debates about coverage metrics.

It's gotta be a whole thing to even think about how to accurately test this kind of software. Simulating all kinds of hardware failures, network partitions, power failures, or the thousand other failure modes.

Then again they get like $100B in revenue that should buy some decent unit tests.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#166

Post by voidfunc »

tootie wrote: post #162
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

Second hand info but supposedly when an outage hits they go all hands on resolving it and no one who knows what's going on has time to update the status board which is why it's always behind.

Not AWS, but Azure: highly doubt. At least at Azure the moment you declare an outage there is a incident manager to handle customer communication.

Bullshit someone at Amazon doesn’t have time to update the status.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#167

Post by EdwardDiego »

shagie wrote: post #160
Fordec wrote: post #63

Earlier quoted context omitted.

Simply already patched. Company sizes and number of attack surfaces vary. 22 hours is plenty of time for an input string filter on a centrally controlled endpoint and a dependency increment with the right CI pipeline.

Consider the possible ways for a string to be injected into any of the following: Apache Solr Apache Druid Apache Flink ElasticSearch Flume Apache Dubbo Logstash Kafka If you've got any of them, they're likely exploitable too. That list comes from: https://unit42.paloaltonetworks.com/apache-log4j-vulnerabili... The attack surface is quite a bit larger than many realize. I recently had a conversation with a person who…

Kafka is still on log4j1. It's only vulnerable if you're using a JMSAppender.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#168

Post by Fordec »

shagie wrote: post #160
Fordec wrote: post #63

Earlier quoted context omitted.

Simply already patched. Company sizes and number of attack surfaces vary. 22 hours is plenty of time for an input string filter on a centrally controlled endpoint and a dependency increment with the right CI pipeline.

Consider the possible ways for a string to be injected into any of the following: Apache Solr Apache Druid Apache Flink ElasticSearch Flume Apache Dubbo Logstash Kafka If you've got any of them, they're likely exploitable too. That list comes from: https://unit42.paloaltonetworks.com/apache-log4j-vulnerabili... The attack surface is quite a bit larger than many realize. I recently had a conversation with a person who…

Lets see...

Nope. Nope. Nope. Nope. Nope. Nope. Nope.

aaaand...

Nope. Plans for it, but not yet in production.

Oh and before anyone starts, not in transitive dependencies either. Just good old bare metal EC2 instances without vendor lock in.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#169

Post by nijave »

propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

Over two years I think we'd see about 2-3 AZ issues but only once I would consider it an outage.

Usually there would be high network error rates which were usually enough to make RDS Postgres fail over if it was in the impacted AZ

The only real "outage" was DNS having extremely high error rates in a single us-east-1 AZ to the point most things there were barely working

Lack of instance capacity, especially spot, especially for the NVMe types was common of CI (it used ASGs for builder nodes). It'd be pretty common for a single AZ to run out of spot instance types--especially the NVMe ([a-z]#d types)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#170

Post by sbierwagen »

bamboozled wrote: post #95
a45a33s wrote: post #41

Earlier quoted context omitted.

how are auth requests supposed to reach the auth server if the networking is broken?

I’d accept this as an answer if I received a timeout or a message to say that. Permission denied is something altogether because it implies the request reached an authorisation system, was evaluated and denied.

Fail-secure + no separate error for timeouts maybe? If the server can't be reached then it just denies the request.
Post reply on HN