Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS's us-east-1 region is experiencing issues

health.aws.amazon.com

Post reply on HN
Print view View original

41–50 of 168 posts

Re: AWS's us-east-1 region is experiencing issues

#41

Post by fragmede »

halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

You're right, but that's been true since the beginning of the tech boom (but isn't exclusive to tech) when no one works for a place for several decades. Companies weather this in different ways but attrition has always been around.

What's causing people to believe that the latest round of attrition is any different?

Re: AWS's us-east-1 region is experiencing issues

#42

Post by fragmede »

frays wrote: post #38
operator1 wrote: post #11

What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?

Increase in attrition across the industry. A lot of institutional knowledge in these massive tech corporations is disappearing and we're starting to reach the tipping point.

But there's always been attrition. What are some of the ways that is now different that is affecting attrition rates and their effects?

Re: AWS's us-east-1 region is experiencing issues

#43

Post by ElevenLathe »

saltypal wrote: post #29
nostrebored wrote: post #24

Earlier quoted context omitted.

It definitely is. For an issue like this, you will see relevant teams and delegates looped in very quickly. Getting approved wording about an outage requires some very senior people though. Often they have to be paged in as well. Having worked at a few other large tech companies now -- Amazon's incident response process is honestly great. It's one of the things I miss about working there.

This. We have a 4-person team and posted our own incident about this 7 minutes before Amazon did. Surely they can aim a little higher.

IME, this actually becomes more challenging as a company gets larger, not less (but that doesn't mean it can't be done).

Re: AWS's us-east-1 region is experiencing issues

#44

Post by ranman »

tyingq wrote: post #18
xilni wrote: post #8

This is why you are strongly urged not to rely on one region or AZ.

Good advice, though AWS still has some services that don't work completely independently. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And I wish they didn't have global-looking endpoints (like https://sts.amazonaws.com ) that aren't really global or resilient.

STS will let you use regional endpoints now, right?

Re: AWS's us-east-1 region is experiencing issues

#45

Post by nyellin »

halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

That's the problem we're out to solve with robusta.dev.

We're slowly but surely converting the world's institutional technical knowledge into re-usable and automated runbooks.

Re: AWS's us-east-1 region is experiencing issues

#46

Post by adamrezich »

xeromal wrote: post #36
operator1 wrote: post #11

What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?

Russian war is another juicy possibility

told myself I'd click this submission's comments link, CTRL+F `Russia`, & quit HN for the day if anything came up, thanks for not disappointing

Re: AWS's us-east-1 region is experiencing issues

#47

Post by quxbar »

300bps wrote: post #30
operator1 wrote: post #11

What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?

Important to keep in mind that AWS has 250 services in 84 Availability Zones in 26 regions. This outage is reportedly impacting 5 services in 1 region. For those impacted, pretty terrible. But as a heavy user of AWS, I’ve seen these notices posted multiple times on HN and haven’t been impacted by one yet.

For businesses with uptime guarantees and lots of boxes to spin up in failover scenario, this has been a very eventful 12 months. At least that's what I'm experiencing.

Re: AWS's us-east-1 region is experiencing issues

#48

Post by xeromal »

adamrezich wrote: post #46
xeromal wrote: post #36

Earlier quoted context omitted.

Russian war is another juicy possibility

told myself I'd click this submission's comments link, CTRL+F `Russia`, & quit HN for the day if anything came up, thanks for not disappointing

Haha, no problemo.

Re: AWS's us-east-1 region is experiencing issues

#49

Post by stone-monkey »

fragmede wrote: post #42
frays wrote: post #38

Earlier quoted context omitted.

Increase in attrition across the industry. A lot of institutional knowledge in these massive tech corporations is disappearing and we're starting to reach the tipping point.

But there's always been attrition. What are some of the ways that is now different that is affecting attrition rates and their effects?

Probably increased salary and switch to permanent remote. Amazon is notorious for their frugality and they recently doubled their maximum salary cap to 350k. They would only have done this to stay competitive in the current job market. This implies that many of their existing employees are underpaid relative to their peers at comparable companies and they've likely seen a large uptick in attrition. Not to mention attrition begets more attrition, especially if it's "influential" employees who are leaving.

Re: AWS's us-east-1 region is experiencing issues

#50

Post by tyingq »

ranman wrote: post #44
tyingq wrote: post #18

Earlier quoted context omitted.

Good advice, though AWS still has some services that don't work completely independently. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And I wish they didn't have global-looking endpoints (like https://sts.amazonaws.com ) that aren't really global or resilient.

STS will let you use regional endpoints now, right?

Yes. It's just that the "global endpoint" is misleading. They don't repoint it if it fails. It really shouldn't exist given that's how it functions.
Post reply on HN