AWS us-east-1 outage • Hacker News Board

Re: AWS us-east-1 outage

#421

Post by nightpool » Tue, Dec 07, 2021, 5:34 PM UTC

ballenf wrote: post #199

zegl wrote: post #13

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

https://downdetector.com Amazing and scary to see all the unrelated services down right now.

I think it's pretty unlikely that both Google and Facebook are affected by this minor AWS outage, whatever DownDetector says. I even did a spot check on some of the smaller websites they report as "down", like canva.com, and didn't see any issues.

Re: AWS us-east-1 outage

#422

Post by iso1210 » Tue, Dec 07, 2021, 5:35 PM UTC

jaywalk wrote: post #412

iso1210 wrote: post #30

My raspberry pi is still working just fine

Cool. Now let's have a race to see who can triple their capacity the fastest. (Note: I don't use AWS, so I can actually do it)

Why would I want to triple my capacity?

Most people don't need to scale to a billion users overnight.

Re: AWS us-east-1 outage

#423

Post by arch-ninja » Tue, Dec 07, 2021, 5:35 PM UTC

smt88 wrote: post #62

endisneigh wrote: post #31

Azure, Google Cloud, AWS and others need to have a “Status alliance” where they determine the status of each of their services by a quorum using all cloud providers. Status pages are virtually useless these days

They can do this without an alliance. They very intentionally choose not to do it. Every major company has moved away from having accurate status pages.

Steam has a great status page, companies like that and cloudfare will eat Alphabet's lunch in the next 17-18 years.

Re: AWS us-east-1 outage

#424

Post by ComputerGuru » Tue, Dec 07, 2021, 5:36 PM UTC

adamtester wrote: post #205

PeterBarrett wrote: post #71

Earlier quoted context omitted.

I hope you stay being the only person who has said that, 1 region being gone is enough for me!

I should have said, only the Console and CLI was down for us, our services remained up!

Console has lots of us-east-1 dependencies.

Re: AWS us-east-1 outage

#425

Post by AzzieElbab » Tue, Dec 07, 2021, 5:36 PM UTC

Betting on dynamo again

Re: AWS us-east-1 outage

#426

Post by mountainofdeath » Tue, Dec 07, 2021, 5:37 PM UTC

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

Former AWSser. I can totally believe that happened and continues to happen in some teams. Officially, it's not supposed to be done that way.

Some AWS managers and engineers bring their corporate cultural baggage with them when they join AWS and it takes a few years to unlearn it.

Re: AWS us-east-1 outage

#427

Post by numberwhun » Tue, Dec 07, 2021, 5:38 PM UTC

Amazon is having an outage in us-east-1, and it is bleeding over elsewhere, like eu: https://status.aws.amazon.com/

Re: AWS us-east-1 outage

#428

Post by zrail » Tue, Dec 07, 2021, 5:39 PM UTC

whoisjuan wrote: post #403

us-east-1 is so unreliable that it probably should be nuked. It's the region with the worst track record. I guess it doesn't help that is one of the oldest.

US-EAST-1 is the oldest, largest, and most heterogeneous region. There are many data centers and availability zones within the region. I believe it's where AWS rolls out changes first, but I'm not confident on that.

Re: AWS us-east-1 outage

#429

Post by skwirl » Tue, Dec 07, 2021, 5:39 PM UTC

blahyawnblah wrote: post #163

temuze wrote: post #113

Friends tell friends to pick us-east-2. Virginia is for lovers, Ohio is for availability.

Lots of services are only in us-east-1. The sso system isn't working 100% right now so that's where I assume it's hosted.

Yeah, there are "global" services which are actually secretly us-east-1 services as that is the region they use for internal data storage and orchestration. I can't launch instances with OpsWorks (not a very widely used service, I'd imagine) even if those instances are in stacks outside of us-east-1. I suspect Route53 and CloudFront will also have issues.

Re: AWS us-east-1 outage

#430

Post by avaika » Tue, Dec 07, 2021, 5:40 PM UTC

LinuxBender wrote: post #304

endisneigh wrote: post #31

Azure, Google Cloud, AWS and others need to have a “Status alliance” where they determine the status of each of their services by a quorum using all cloud providers. Status pages are virtually useless these days

Or just modify sites like DownDetector to show who is hosting each site. When{n} number of sites are down hosted on {x} one could draw a conclusion. It won't be as detailed as "xyz services failed" but rather the overall operational chain is broken. There could be a graph that shows 99% of sites hosted on amazon US East 1 down it would be hard to hide that. This could also paint a picture of what companies are not ac…

Cloud providers are not just about web hosting. There are dozens of tools hidden from end users. E.g. services used for background operations / maintenance (e.g. aws codecommit / codebuild / or even aws web console like today). This kind of outages won't bring down your web site, but still might break your normal workflow and even cost you some money.