Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

641–650 of 1001 posts

Re: AWS us-east-1 outage

#641

Post by Frost1x »

andrewguenther wrote: post #620
mrsuprawsm wrote: post #465

Earlier quoted context omitted.

If your statement is true, then why is the AWS status page widely considered useless, and everyone congregates on HN and/or Twitter to actually know what's broken on AWS during an outage?

> Yes, VP approval is needed to make any updates on the status dashboard. But that's not as hard as it may seem. AWS executives are extremely operation-obsessed, and when there is an outage of any size are engaged with their service teams immediately. My experience generally aligns with amzn-throw, but this right here is why. There's a manual step here and there's always drama surrounding it. The process to update th…

It's intentional plausible deniability. By creating the manual step you can shift blame away. It's just like the concept of personal health dashboards which are designed to keep an asymmetry in reliability information from a host and the client to their personal anecdata experiences. Ontop of all of this, the metrics are pretty arbitrary.

Let's not pretend businesses haven't been intentionally advertising in deceitful ways for decades if not hundreds of years. This just happens to be current strategy in tech of lying and deceiving customers to limit liability, responsibility, and recourse actions.

To be fair, it's not it's not just Amazon, they just happen to be the largest and targeted whipping boys on the block. Few businesses under any circumstances will admit to liability under any circumstances. Liability has to always be assessed externally.

Re: AWS us-east-1 outage

#644

Post by pm90 »

cwkoss wrote: post #572
dentemple wrote: post #184

Earlier quoted context omitted.

It's because none of these companies are held responsible for missing their actual SLAs, as opposed to their self-reported SLA compliance. So unless regulation gets implemented that says otherwise, there's zero incentive for any company to maintain an accurate status page.

I wonder if there could be profitable play where an organization monitors SLA compliance, and then produces a batch of lawsuits or class action suit on behalf of all of its members when the SLA is violated.

This is a neat idea. Install a simple agent in all customers' environments, select AWS dependencies, then monitor uptime over time. Aggregate across customers, and then go to AWS with this data.

Re: AWS us-east-1 outage

#645

Post by shepherdjerred »

nostrebored wrote: post #317
itsyaboi wrote: post #289

Earlier quoted context omitted.

Slack, as of last year. https://slack.com/blog/news/slack-aws-drive-development-agil...

And before that, Amazon Chime was the messaging and conferencing tool. Now that I'm not using it, I actually miss it a lot!

I cried tears of joy when Amazon finally switched to Slack last year

Re: AWS us-east-1 outage

#646

Post by jaywalk »

iso1210 wrote: post #615
jaywalk wrote: post #506

Earlier quoted context omitted.

Many B2B-type applications have a lot of usage during the workday and minimal usage outside of it. No reason to keep all that capacity running 24/7 when you only need most of it for ~8 hours per weekday. The cloud is perfect for that use case.

Is it really? How much does that scaling actually cost? And what's a workday anyway, surely you operate globally?

Scaling itself costs nothing, but saves money because you're not paying for unused capacity.

The main application I run operates in 7 countries globally, but the US is the only one that has enough usage to require additional capacity during the workday. So out of 720 hours in a 30 day month, cloud scaling allows me to pay for additional capacity for only the (roughly) 160 hours that it's actually needed. It's a significant cost saver.

And because the scaling is based on actual metrics, it won't scale up on a holiday when nobody is using the application. More cost savings.

Re: AWS us-east-1 outage

#647

Post by oscribinn »

amzn-throw wrote: post #444
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

100 BEZOBUCKS™ have been deposited to your account for this post.

Re: AWS us-east-1 outage

#649

Post by foobiekr »

ssimpson wrote: post #315
javajosh wrote: post #302

Earlier quoted context omitted.

I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…

when working on CloudFiles, we often had monitoring for our limited dependencies that were better than their monitoring. Don't just know what your stuff is doing, but what your whole dependency ecosystem is doing and know when it all goes south. also helps to learn where and how you can mitigate some of those dependencies.

This. We found very big, serious issues with our anti-DDOS provider because their monitoring sucked compared to ours. It was a sobering reality check when we realized that.

Re: AWS us-east-1 outage

#650

Post by mynameisvlad »

res0nat0r wrote: post #577
mynameisvlad wrote: post #544

Earlier quoted context omitted.

SLAs. Officially acknowledging an incident means that they now have to issue the SLA credits.

The outage dashboard is normally only updated if a certain $X percent of hosts / service is down. If the EC2 section were updated every time a rack in a datacenter went down, it would be red 24x7. It's only updated when a large percentage of customers are impacted, and most of the time this number is less than what the HN echo chamber makes it appear to be.

I mean, sure, there are technical reasons why you would want to buffer issues so they're only visible if something big went down (although one would argue that's exactly what the "degraded" status means).

But if the official records say everything is green, a customer is going to have to push a lot harder to get the credits. There is a massive incentivization to “stay green”.

Post reply on HN