Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

561–570 of 1001 posts

Re: AWS us-east-1 outage

#561

Post by alexatalktome »

mystcb wrote: post #213

Not sure it helps, but got this update from someone inside AWS a few moments ago. "We have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also services like S3. We are actively working towards recovery."

Internally the rumor is that our CICD pipelines failed to stop bad commits to certain AWS services. This isn’t due to tests but due to actual pipelines infra failing.

We’ve been told to disable all pipelines even if we have time blockers or manual approval steps or failing tests

Re: AWS us-east-1 outage

#562

Post by jrd259 »

yaacov wrote: post #509
amzn-throw wrote: post #444

Earlier quoted context omitted.

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

Can’t comment on most of your post but I know a lot of Amazon engineers who think of the CoE process (Correction of Error, what other companies would call a postmortem) as punitive

I don't know any, and I have written or reviewed about 20

Re: AWS us-east-1 outage

#563

Post by ARandomerDude »

abledon wrote: post #414
kumarakn wrote: post #336

Earlier quoted context omitted.

I worked at Walmart Technology. I bravely wrote post mortem documents owning the fault of my team (100+ people), owning both technically and also culturally as their leader. I put together a plan to fix it and executed it. Thought that was the right thing to do. This happend two times in my 10 year career there. Both times I was called out as a failure in my performance eval. Second time, I resigned and told them to…

is it just 'ceremony' to be called out on those things? (even if it is actually a positive sum total)

> Happy now I am out of such shitty place.

Doesn't sound like it.

Re: AWS us-east-1 outage

#565

Post by [deleted] »

dangrossman wrote: post #516
mabbo wrote: post #66

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?

My website that runs on US-East-1 is up. However, my Alexa (Echo) won't control my thermostat right now. And my Ring app won't bring up my cameras. Those services are run on AWS.

[deleted]

Re: AWS us-east-1 outage

#566

Post by the-pigeon »

amzn-throw wrote: post #444
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

What?!

Everybody is very slow to update their outage pages because of SLAs. It's in a company's financial interest to deny outages and when they are undeniable to make them appear as short as possible. Status pages updating slowly is definitely by design.

There's no large dev platform I've used that this wasn't true of their status pages.

Re: AWS us-east-1 outage

#567

Post by jrockway »

unionpivo wrote: post #491
bandyaboot wrote: post #415

Earlier quoted context omitted.

Where does this mindset end? Do I lack due diligence by choosing to accept that the cpu microcode on the system I’m deploying to works correctly?

If it's brand new RiscV CPU that was just relesed 5 min ago, and nobody really tested then yes. If its standard CPU that everybody else uses, and its not known to be bad then no. Same for software. Is it ok to have dependency on AWS services ? Their history shows yes. Dependency on brand new SaaS product ? Nothing mission critical. Or npm/crates/pip packages. Packages that have been around and seedily maintained for…

Why? Intel has Spectre/Meltdown which erased like half of everyone's capacity overnight.

Re: AWS us-east-1 outage

#569

Post by dhsigweb »

dekhn wrote: post #443
jedberg wrote: post #176

Earlier quoted context omitted.

Like I said I never worked there and this is all hearsay but there is a lot of nuance here being missed like partial outages.

This is no longer a partial outage. The status page reports elevated API error rates, DynamoDB issues, EC2 API error rates, and my company's monitoring is significantly affected (IE, our IT folks can't tell us what isn't working) and my AWS training class isn't working either. If this needed a CEO to eventually get around to pressing a button that said "show users the actual information about a problem" that reflects…

My friend works at a telemetry company for monitoring and they are working on alerting customers of cloud service outages before the cloud providers since the providers like to sit on their hands for a while (presumably to try and fix it before anyone notices).

Re: AWS us-east-1 outage

#570

Post by AtlasBarfed »

ctvo wrote: post #528
AtlasBarfed wrote: post #514

Earlier quoted context omitted.

Because OTHERWISE people might think AMAZON is a DYSFUNCTIONAL company that is beginning to CRATER under its HORRIBLE work culture and constant H/FIRE cycle. See, AWS is basically turning into a long standing utility that needs to be reliable. Hey, do most institutions like that completely turn over their staff every three years? Yeah, no. Great for building it out and grabbing market share. Maybe not for being the b…

> Maybe not for being the basis of a reliable substrate of the modern internet. Maybe THEY will go to a COMPETITOR and THINGS MOVE ON if it's THAT BAD. I wasn't sure what the pattern for all caps was, so just giving it a shot there. Apologies if it's incorrect.

I was mocking the parent, who was doing that. Yes it's awful. Effective? Sigh, yes. But awful.
Post reply on HN