Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: AWS us-east-1 outage

#681

Post by hvgk »

AnIdiotOnTheNet wrote: post #673
hvgk wrote: post #587

Well I got to bugger off home early so good job Amazon. Edit: to be clear this is because I’m utterly helplessly unable to do anything at the moment.

Yep, that's a consideration of going with cloud tech: if something goes wrong you're often powerless. At least with on-prem you know who to wake up in the middle of the night and you'll get straight-forward answers about what's going on.

Depends which provider you host your crap with. I’ve had real trouble trying to get a top tier incident even acknowledged by one of the pre cloud providers.

To be fair when it’s AWS when something goes snap it’s not my problem which I’m happy about (until some wise ass at AWS hires me) :)

Re: AWS us-east-1 outage

#682

Post by politician »

winrid wrote: post #678
s_dev wrote: post #666

Earlier quoted context omitted.

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

In this case, just connect over LAN.

Or BlueTooth.

Re: AWS us-east-1 outage

#683

Post by [deleted] »

tekromancr wrote: post #588
jesboat wrote: post #484

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

I just want to serve 5 terabytes of data

[deleted]

Re: AWS us-east-1 outage

#686

Post by muvb00 »

temp6363t wrote: post #598
gunapologist99 wrote: post #492

Earlier quoted context omitted.

That's shockingly stupid. I also worked for a major Walmart IT services vendor in another life, and we always had to be careful about how we handled them, because they didn't always show a lot of respect for vendors. On another note, thanks for building some awesome stuff -- walmart.com is awesome. I have both Prime and whatever-they're-currently-calling Walmart's version and I love that Walmart doesn't appear to mix…

walmart.com user design sucks. My particular grudge right now is - I'm shopping to go pickup some stuff (and indicate "in store pickup) and each time I search for the next item, it resets that filter making me click on that filter for each item on my list.

Walmart.com, Am I the only one in the world who can't view their site on my phone? I tried it on a couple devices and couldn't get it to work. Scaling is fubar. I assumed this would be costing them millions/billions since it's impossible to buy something from my phone right now. S21+ in portrait on multiple browsers.

Re: AWS us-east-1 outage

#687

Post by s_dev »

winrid wrote: post #678
s_dev wrote: post #666

Earlier quoted context omitted.

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

In this case, just connect over LAN.

Right -- I think I've misread OP as graceful fallback e.g. working offline.

Rather than implement a dynamically switching backup in the event of AWS going down which is not trivial.

Re: AWS us-east-1 outage

#688

Post by dekhn »

dekhn wrote: post #440
dekhn wrote: post #146

Earlier quoted context omitted.

if you cannot access the control plane to create or destroy resources, it is down (partial availability). The jobs that are running are basically zombies.

I'm right in the middle of an AWS-run training and we literally can't run the exercises because of this. let me repeat that: my AWS trainign that is run by AWS that I pay AWS for isn't working, because AWS is having control plane (or other) issues. This is several hours after the initial incident. We're doing training in us-west-2, but the identity service and other components run in us-east-1.

We ran through the whole 4.5 hour training and the training app didn't work the entire time.

Re: AWS us-east-1 outage

#689

Post by jaywalk »

vp8989 wrote: post #668
jaywalk wrote: post #646

Earlier quoted context omitted.

Scaling itself costs nothing, but saves money because you're not paying for unused capacity. The main application I run operates in 7 countries globally, but the US is the only one that has enough usage to require additional capacity during the workday. So out of 720 hours in a 30 day month, cloud scaling allows me to pay for additional capacity for only the (roughly) 160 hours that it's actually needed. It's a signi…

You are (conveniently or not) incorrectly assuming that the unit price of provisioned vs on-demand capacity is the same. It's not.

Nice of you to assume that I don't understand the pricing of the services I use. I can assure you that I do, and I can also assure you that there is no such thing as provisioned vs on-demand pricing for Azure App Service until you get into the higher tiers. And even in those higher tiers, it's cheaper for me to use on-demand capacity.

Obviously what I'm saying will not apply to all use cases, but I'm only talking about mine.

Re: AWS us-east-1 outage

#690

Post by threecheese »

miken123 wrote: post #460
amzn-throw wrote: post #444

Earlier quoted context omitted.

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

Well, the narrative is sort of what Amazon is asking for, heh? The whole us-east-1 management console is gone, what is Amazon posting for the management console on their website? "Service degradation" It's not a degradation if it's outright down. Use the red status a little bit more often, this is a "disruption", not a "degradation".

I'm part of a large org with a large AWS footprint, and we've had a few hundred folks on a call nearly all day. We have only a few workloads that are completely down; most are only degraded. This isn't a total outage, we are still doing business in east-1. Is it "red"? Maybe! We're all scrambling to keep the services running well enough for our customers.
Post reply on HN