Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

361–370 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#361

Post by sponaugle »

divbzero wrote: post #256

> This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it. Disruption of the standard incident response mechanism seems to be a common element of longer lasting incidents.

Indeed - Even the recent facebook outage outlined how slow recovery can be if the primary investigation and recovery methods are directly impacted as well. Back in the old days some environments would have POTS dial-in connections to the consoles as backup for network problems. That of course doesn't scale, but it was an attempt to have an alternate path of getting to things. Regrettably if a backhoe takes out all of the telecom at once that plan doesn't work so well.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#362

Post by swasheck »

theneworc wrote: post #358
jrochkind1 wrote: post #344

Earlier quoted context omitted.

> I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message. I gotta say, the implication that you can't register an outage until you know why it happened is pretty damning. The status page is where we look to see if services are effected, if that information can't be shared there until you understand the cause, that's very broken. The AWS status page has becom…

Can you please help me understand why you, and everyone else, are so passionate about the status page? I get that it not being updated is an annoyance, but I cannot figure out why it is the single most discussed thing about this whole event. I mean, entire services were out for almost an entire day, and if you read HN threads it would seem that nobody even cares about lost revenue/productivity, downtime, etc. The vas…

aws isn’t a hobby platform. businesses are built on aws and other cloud providers. those businesses customers have the expectation of knowing why they are not receiving the full value of their service.

it makes sense that part of marketing yourself as a viable infrastructure upon which other businesses can operate, you’d provide more granular and refined communication to allow better communication up and down the chain instead of forcing your customers to rca your service in order to communicate to their customers.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#363

Post by sponaugle »

"Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event. "

That is an interesting way to phrase that. A 'well-tested' method, but 'latent issues'. That would imply the 'well-tested' part was not as well-tested as it needed to be. I guess 'latent issue' is the new 'bug'.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#365

Post by waz0wski »

xyst wrote: post #292

I am not a fan of AWS due to their substantial market share on cloud computing. But as a software engineer I do appreciate their ability to provide fast turnarounds on root cause analyses and make them public.

This isn't a good example of an RCA - as other commenters have noted, it's outrightly lying about some issues during the incident, and using creative language to dance around other problems many people encountered.

If you want to dive into postmortems, there are some repos linking other examples

https://github.com/danluu/post-mortems

https://codeberg.org/hjacobs/kubernetes-failure-stories

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#366

Post by aflag »

theneworc wrote: post #358
jrochkind1 wrote: post #344

Earlier quoted context omitted.

> I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message. I gotta say, the implication that you can't register an outage until you know why it happened is pretty damning. The status page is where we look to see if services are effected, if that information can't be shared there until you understand the cause, that's very broken. The AWS status page has becom…

Can you please help me understand why you, and everyone else, are so passionate about the status page? I get that it not being updated is an annoyance, but I cannot figure out why it is the single most discussed thing about this whole event. I mean, entire services were out for almost an entire day, and if you read HN threads it would seem that nobody even cares about lost revenue/productivity, downtime, etc. The vas…

A status page give you confidence that the problem indeed lies with Amazon and not your own software. I don't think it's very reasonable to notice issues, ask other teams if they are also having issues, and if so, just shrug it off and get a cup of coffee without more investigation. Just because it looks like the problem is with AWS, you can't be sure until you further investigate it, specially if the status page says it's all working fine.

I think it goes without saying that having an outage is bad, but having an outage which is not confirmed by the service provider is even worse. People complain about that a lot because it's the least they could do.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#367

Post by filoleg »

ignoramous wrote: post #313
hericium wrote: post #269

Earlier quoted context omitted.

I assumed in good faith that this is someone knowing internals as a larger customer, not an AWS person shit-talking other AWS teams. Got curious only after a downvote hence late edit. My bad.

> ...an AWS person shit-talking other AWS teams [in public]. I remember a time when this would be an instant reprimand... Either amzn engs are bolder these days, or amzn hr is trying really hard for amzn to be "world's best employer", or both.

Gotta deanonymize the user to reprimand them. Maybe i am wrong here, but i don’t see it as something an Amazon HR employee would actually waste their time on (exceptions apply for confidential info leaks and other blatantly illegal stuff, of course). Especially given that it might as well be impossible, unless the user incriminated themselves with identifiable info.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#368

Post by chakspak »

gingerlime wrote: post #289
amzn-throw wrote: post #285

Earlier quoted context omitted.

> a VP must sign off on changing status pages, which is... backwards to say the least. I think most people's experience with "VP's" makes them not realize what AWS VP's do. VP's here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to "Call me when the data center is back up and running again!" They are on the tech call, working with the engineers, evaluating…

> I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message. I think that's the crux of the matter? AWS seems to now have a reputation for ignoring issues that are easily observable by customers, and by the time any update shows up, it's way too late. Whether VPs make this decision or not is irrelevant. If this becomes a known pattern (and I think it has), then…

> disclaimer: I have very little skin in this game. We use S3 for some static assets, and with layers of caching on top, I think we are rarely affected by outages. I'm still curious to observe major cloud outages and how they are handled, and the HN reaction from people on both side of the fence.

I'd like to share my experience here. This outage definitely impacted my company. We make heavy use of autoscaling, we use AWS CodeArtifact for Python packages, and we recently adopted AWS Single Sign-On and EC2 Instance Connect.

So, you can guess what happened:

- No one could access the AWS Console.

- No one could access services authenticated with SAML.

- Very few CI/CD, training or data pipelines ran successfully.

- No one could install Python packages.

- No one could access their development VMs.

As you might imagine, we didn't do a whole lot that day.

With that said, this experience is unlikely to change our cloud strategy very much. In an ideal world, outages wouldn't happen, but the reason we use AWS and the cloud in general is so that, when they do happen, we aren't stuck holding the bag.

As others have said, these giant, complex systems are hard, and AWS resolved it in only a few hours! Far better to sit idle for a day rather than spend a few days scrambling, VP breathing down my neck, discovering that we have no disaster recovery mechanism, and we never practiced this, and hardware lead time is 3-5 weeks, and someone introduced a cyclical bootstrapping process, and and and...

Instead, I just took the morning off, trusted the situation would resolve itself, and it did. Can't complain. =P

I might be more unhappy if we had customer SLAs that were now broken, but if that was a concern, we probably should have invested in multi-region or even multi-cloud already. These things happen.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#369

Post by gnabgib »

propter_hoc wrote: post #27
dijit wrote: post #13

Earlier quoted context omitted.

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

I agree with you, but my services are actually in Canada (Central). There's only one region in Canada, so I don't really have an alternative. AWS justifies it by saying there are three AZs (distinct data centres) within Canada (Central), but I get scared when I see these region-wide issues. If the AZs were really distinct, you wouldn't really have region-wide issues.

Good news, a new region is coming to Canada in the west[0] eta 2023/24

[0]: https://aws.amazon.com/blogs/aws/in-the-works-aws-canada-wes...

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#370

Post by pfortuny »

gfodor wrote: post #266
colechristensen wrote: post #237

Earlier quoted context omitted.

> System dynamics are hard. And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.

Why is this hard, and can’t just be written down somewhere as part of the engineering discipline? This aspect of systems in 2021 really shouldn’t be an “art.”

Exponential behaviour is hard to understand.

Also, experiments at this size and speed have never been carried out before.

And statistical behaviours are very difficult to understand. First thing: 99.9999% uptime for ALL users is HUGELY different from 99.999%.

As a matter of fact, this was just one of amazon’s zones, rememeber.

Edit: finally, the right model for these systems might well have no mean (fat tails…) and then where do the statistics go from there?

Post reply on HN