Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

721–730 of 1001 posts

Re: AWS us-east-1 outage

#721

Post by codegeek »

dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

"If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work."

Building fallbacks require work. How much extra effort and overhead is needed to build something like this ? Sometimes the cost vs benefits says that it is ok not to do it. If AWS has an outage like this once a year, maybe we can deal with it (unless you are working with mission critical apps).

Re: AWS us-east-1 outage

#722

Post by mro_name »

9wzYQbTYsAIc wrote: post #703
thayne wrote: post #700

Earlier quoted context omitted.

there's a reason it is called the Internet of things, and not the "local network of things". Even if the latter is probably what most customers would prefer.

There’s also no reason for an internet connected app to crash on load when there is no access to the internet services.

indeed.

A constitutional property of a network is it's volatility. Nodes may fail. Edges may. You may not. Or you may. But then you're delivering no reliabilty but crap. Nice sunshine crap, maybe.

Re: AWS us-east-1 outage

#723

Post by stevehawk »

rehevkor5 wrote: post #705
s_dev wrote: post #666

Earlier quoted context omitted.

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

Usually this refers to falling back to a different region in AWS. It's typical for systems to be deployed in multiple regions due to latency concerns, but it's also important for resiliency. What you call "a very small edge case" is occurring as we speak, and if you're vulnerable to it you could be losing millions of dollars.

probably not possible for a lot mroe services than you'd think because AWS Cognito has no decent failover method

Re: AWS us-east-1 outage

#724

Post by codegeek »

amzn-throw wrote: post #444
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

"Yes, VP approval is needed to make any updates on the status dashboard."

If services are clearly down, why is this needed ? I can understand the oversights required for a company like Amazon but this sounds strange to me. If services are clearly down, I want that damn status update right away as a customer.

Re: AWS us-east-1 outage

#725

Post by milofeynman »

lordnacho wrote: post #56

This got me thinking, are there any major chat services that would go down if a particular AWS/GCP/etc data centre went down? You don't want your service to go down, plus your team's comms at the same time.

Remember when Facebook went down? Fb, Whatsapp, messenger, Instagram were all down. Don't know what they use internally

Re: AWS us-east-1 outage

#726

Post by stevehawk »

jesboat wrote: post #484
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

I don't know if that should surprise us. AWS hosted their status page in S3 so it couldn't even reflect its own outage properly ~5 years ago. https://www.theregister.com/2017/03/01/aws_s3_outage/

Re: AWS us-east-1 outage

#727

Post by marcinzm »

amzn-throw wrote: post #444
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

>* Every AWS customer has a PERSONAL health dashboard in the console that should indicate their experience.

You mean the one that is down right now?

Re: AWS us-east-1 outage

#728

Post by dylan604 »

crad wrote: post #65

We make heavy usage of Kinesis Firehose in us-east-1. Issues started ~1:24am ET and resolved around 7:31am ET. Then really kicked in at a much larger scale at 10:32am ET. We're now seeing failures with connections to RDS Postgres and other services. Console is completely unavailable to me.

>Issues started ~1:24am ET and resolved around 7:31am ET.

First engineer found a clever hack using bubble gum

>Then really kicked in at a much larger scale at 10:32am ET.

Bubble gum dried out, and the connector lost connection again. Now, connector also fouled by the gum making a full replacement required.

Re: AWS us-east-1 outage

#729

Post by organsnyder »

lsaferite wrote: post #580
minig33 wrote: post #409

Earlier quoted context omitted.

I can’t turn on my lights… the future is weird

And that is why my lighting automation has a baseline req that it works 100% without the internet and preferably without a central controller.

I love my Home Assistant setup for this reason. I can even get light bulbs pre-flashed with ESPHome now (my wife was bemused when I was updating the firmware on the lightbulbs).

Re: AWS us-east-1 outage

#730

Post by dylan604 »

beamatronic wrote: post #717
lordnacho wrote: post #711

Earlier quoted context omitted.

If you did that some clever person would set up their PiHole so that their device just always worked, and then you couldn't send them ads and surveil them. They'd tell their friends and then everyone would just use their local devices locally. Totally irresponsible what you're suggesting.

An even more clever person would package up this box, and sell it, along with a companion subscription service, to help busy folks like myself.

But this new little box would then be required to connect to the home server to receive updates. Guess what? No updates, no worky!! It's a vicious circle!!! Outages all the way down
Post reply on HN