Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS's us-east-1 region is experiencing issues

health.aws.amazon.com

Post reply on HN
Print view View original

61–70 of 168 posts

Re: AWS's us-east-1 region is experiencing issues

#61

Post by rurp »

SketchySeaBeast wrote: post #37
zwirbl wrote: post #26

Earlier quoted context omitted.

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

If we want to normalize letting long term support people call themselves tech priests I'd very much appreciate it. "What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."

Nothing makes me feel more like a wizard than learning some new command line skills. Just string together some short, seemingly indecipherable symbols, and... magic!

Re: AWS's us-east-1 region is experiencing issues

#63

Post by jedberg »

Protip to anyone building new infrastructure in AWS: If you're gonna only use one region in the US, make it us-east-2 or us-west-2. us-east-1 is their oldest and biggest region but also the least stable in the US (ok technically us-west-1 is worse but you can't get that one anymore).

Re: AWS's us-east-1 region is experiencing issues

#64

Post by thethethethe »

halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

Disc: Googler opinions are my own.

I've been an SRE on a tier 1 GCP product for over three years this is not the case. In my experience, our systems have only gotten more reliable and easier to understand since I joined.

It's not like there are only a few key knowledge holders that single handedly prevent outages from happening. In reality, you don't need to know shit about how a system works internally to prevent outages if things are setup correctly.

In theory, my dog should be able to sit on my laptop and submit a prod breaking change without any fear that it will make it to prod and damage the system because automated testing/canary should catch it and, if it does make it to prod, we should be able to detect and mitigate it before the change effects more users using probes or whitebox monitoring.

This is happens for 99.9% of potential issues and is completely invisible to users. However it's what's not caught (the remaining 0.01%) that actually matters.

Re: AWS's us-east-1 region is experiencing issues

#65

Post by replygirl »

Traster wrote: post #59
halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

I think this is a transient issue. When you're in growth mode you make a huge series of hacks to just keep things running and then when you leave.... well, it's a problem. But if the business is robust, and lives beyond you, what replaces your work is better documented, better tested, and maintainable. That's the dream. Obviously there are companies that sink between v1 and v2, but that's life. Fundamentally I think…

if us-east-1 is "in growth mode", what that we rely on can we possibly expect to ever reach maturity?

Re: AWS's us-east-1 region is experiencing issues

#66

Post by mhio »

saltypal wrote: post #16

Based on our telemetry, this started as NXDOMAINs for sqs.us-east-1.amazonaws.com beginning in modest volumes at 20:43 UTC and becoming a total outage at 20:48 UTC. Naturally, it was completely resolved by 20:57, 5 minutes before anything was posted in the "Personal Health Dashboard" in the AWS console. It takes a while to find a Vice President, I guess.

The truth assuaging usually takes 15-30 minutes.

Re: AWS's us-east-1 region is experiencing issues

#68

Post by thethethethe »

operator1 wrote: post #11

What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?

> What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data.

Source?

Re: AWS's us-east-1 region is experiencing issues

#70

Post by 9wzYQbTYsAIc »

halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

Outside of the mega-fang industry, I’m wondering the same thing.

The Great Resignation had to have taken a huge toll on regular enterprises. There are probably going to be some unlucky (or lucky, depending on how hardcore they are) people in the position of maintaining aging legacy systems and retrofitting them into the future.

COBOL, for example, is becoming a lucrative language for people in the financial and insurance industries. Legacy Java is all over the place, I’m sure. Legacy .NET is in the middle of a huge industry retrofit, (.NET 5 was the official post-legacy rebrand and they’re on to .NET 6+ now).

Post reply on HN