Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

791–800 of 1001 posts

Re: AWS us-east-1 outage

#791

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> AWS (and all other IAAS providers) will beg you to use multiple region

will they? because AWS still puts new stuff in us-east-1 before anywhere else, and there is often a LONG delay before those things go to other regions. there are many other examples of why people use us-east-1 so often, but it all boils down to this: AWS encourage everyone to use us-east-1 and discourage the use of other regions for the same reasons.

if they want to change how and where people deploy, they should change how they encourage it's customers to deploy.

my employer uses multi-region deployments where possible, and we can't do that anywhere nearly as much as we'd like because of limitations that AWS has chosen to have.

so if cloud providers want to encourage multi-region adoption, they need to stop discouraging and outright preventing it, first.

Re: AWS us-east-1 outage

#792

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

They have, it's called hosting on-premesis, and it's even less reliable than cloud providers.

Re: AWS us-east-1 outage

#793

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

they usually beg you to use multiple availability zones though

I'm not sure how many aws services are easy to spawn at multiple regions

Re: AWS us-east-1 outage

#794

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

Maybe has something to do with CloudFront mandating certs to be in us-east-1?

YES! Why do they do that? It's so weird. I will deploy a whole config into us-west-1 or something; but then I need to create a new cert in us-east-1 JUST to let cloudfront answer an HTTPS call. So frustrating.

Re: AWS us-east-1 outage

#795

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

This is the right answer, I recall studying for the solutions architect professional certification and reading this countless times: outages will happen and you should plan for them by using multi-region if you care about downtime.

It's not AWS fault here, it's the companies', which assume that it will never be down. In-house servers also have outages, it's a very naive assumption to think that it'd be all better if all of those services were using their own servers.

Facebook doesn't use AWS and they were down for several hours a couple weeks ago, and that's because they have way better engineers than the average company, working on their infrastructure, exclusively.

Re: AWS us-east-1 outage

#796

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

People are so quick to forget how things were before behemoths like AWS, Google Cloud, and Azure. Not all things are free and the outage the internet is experiencing is the risk users signed up for.

If you would like to go back to the days of managing your own machines, be my guest. Remember those machines also live somewhere and were/are subject to the same BGP and routing issues we've seen over the past couple of years.

Personally, I'll deal with outages a few times a year for the peace of mind that there's a group of really talented people looking into for me.

Re: AWS us-east-1 outage

#797

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

"AWS has fewer outages in one zone alone than the best self-hosted institutions" sure you just call an outage "increased error rate"

Re: AWS us-east-1 outage

#798

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

In addition to fewer outages, _many_ products get a free pass on incidents because basically everyone is being impacted by outage.

Re: AWS us-east-1 outage

#799
post #113

Friends tell friends to pick us-east-2. Virginia is for lovers, Ohio is for availability.

I wonder why AWS has Ohio and Virginia but no region in the northeast where a significant plurality of customers using east regions probably live.

Re: AWS us-east-1 outage

#800
post #788

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. > Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. That's an overblown argument. Nobody is saying that, but it's clear that businesses that maintain their ow…

This feels like a discussion that could sorely use some numbers.

What are good examples of

>a small business running a few websites with a few million hits per month, it might be cheaper and easier to colocate a few servers and hire a few DevOps or old-school sysadmins to administer the infrastructure.

and how often do they go down?

Post reply on HN