Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

141–150 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#141

Post by hunter2_ »

nwallin wrote: post #129
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

So -- ctrl-f "Dash" only produces four results and it's hidden away in the bottom of the page. It's false to claim that even 20% of the post mortem is addressing the failure of the dashboard. The problem is that the dashboard requires VP approval to be updated. Which is broken. The dashboard should be automatic. The dashboard should update before even a single member of the AWS team knows there's something wrong.

Is it typical for orgs (the whole spectrum: IT departments everywhere, telecom, SaaS, maybe even status of non-technical services) to have automatic downtime messaging that doesn't need a human set of eyes to approve it first?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#142

Post by subsaharancoder »

ALittleLight wrote: post #48
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

I worked at AMZN and this perfectly captures my experience there with those weekly reviews. I once set a project I was managing as "Red" and had multiple SDMs excoriate me for apparently "throwing them under the bus" even though we had missed multiple timelines and were essentially not going to deliver anything of quality on time. I don't miss this aspect of AMZN!

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#143

Post by cyounkins »

foobiekr wrote: post #136
cyounkins wrote: post #42

My favorite sentence: "Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event."

thundering herd and accidental synchronization for the win I am sad to say, I find issues like this any time I look at retry logic written by anyone I have not interacted with previously on the topic. It is shockingly common even in companies where networking is their bread and butter.

It absolutely is difficult. A challenge I have seen is when retries are stacked and callers time out subprocesses that are doing retries.

I just find it amusing that they describe their back-off behaviors as "well tested" and in the same sentence, say it didn't back off adequately.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#144

Post by lanstin »

banana_giraffe wrote: post #83
propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

It can be a bit hard to know, since the AZ identifiers are randomized per account, so if you think you have problems in us-west-1a, I can't check on my side. You can get the AZ ID out of your account to de-randomize things, so we can compare notes, but people rarely bother, for whatever reason.

If you do a lot of VPC Endpoints to clients/thirdparties, you learn the AZIDs or you go to all AZIDs in a region by default.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#145

Post by azundo »

> This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues on the devices connecting the two networks.

I remember my first experience realizing the client retry logic we had implemented was making our lives way worse. Not sure if it's heartening or disheartening that this was part of the issue here.

Our mistake was resetting the exponential backoff delay whenever a client successfully connected and received a response. At the time a percentage but not all responses were degraded and extremely slow, and the request that checked the connection was not. So a client would time out, retry for a while, backing off exponentially, eventually successfully reconnect and then after a subsequent failure start aggressively trying again. System dynamics are hard.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#146

Post by markus_zhang »

>At 7:30 AM PST, an automated activity to scale capacity of one of the AWS services hosted in the main AWS network triggered an unexpected behavior from a large number of clients inside the internal network.

Just curious, is this scaling an AWS job or a client job? Looks like an AWS one from the context. I'm wondering if they are deploying additional data centers or something else?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#148

Post by teej »

yegle wrote: post #134

Was this outage only impact us-east-1 region? I think I saw other regions affected in some HN comments but this summary did not mention anything to suggest it has more than 1 region impacted.

There are some AWS services, notably STS, that are hosted in us-east-1. I don’t have anything in us-east-1 but I was completely unable to log into the console to check on the health of my services.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#149

Post by [deleted] »

zeko1195 wrote: post #133
Fordec wrote: post #63

Earlier quoted context omitted.

Simply already patched. Company sizes and number of attack surfaces vary. 22 hours is plenty of time for an input string filter on a centrally controlled endpoint and a dependency increment with the right CI pipeline.

lol no

[deleted]

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#150

Post by ashtonkem »

propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

The best bang for your buck isn’t deploying into multiple AZs, but relocating everything into almost any other region than us-east-1.

My system is latency and downtime tolerant, but I’m thinking I should move all my Kafka processing over to us-west-2

Post reply on HN