Can confirm I can once again login to the Console and everything seems to be back to normal in us-east-2.
AWS us-east-1 outage
821–830 of 1001 posts
Re: AWS us-east-1 outage
#822The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…
"If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work." Building fallbacks require work. How much extra effort and overhead is needed to build something like this ? Sometimes the cost vs benefits says that it is ok not to do it. If AWS has an outage like this once a year, maybe we can deal with it (unless you are working with mission critical apps).
Yes, it is a lot of work to test if response code is OK or not, or if a timeout limit has been reached. So much so, I pretty much wrote the test in the first sentence. Phew. 10x coder right here!
Re: AWS us-east-1 outage
#823Earlier quoted context omitted.
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…
> AWS (and all other IAAS providers) will beg you to use multiple region will they? because AWS still puts new stuff in us-east-1 before anywhere else, and there is often a LONG delay before those things go to other regions. there are many other examples of why people use us-east-1 so often, but it all boils down to this: AWS encourage everyone to use us-east-1 and discourage the use of other regions for the same rea…
It works really well imo. All the people who want to use new stuff at the expense of stability choose us-east-1; those who want stability at the expense of new stuff run multi-region (usually not in us-east-1 )
Re: AWS us-east-1 outage
#824Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…
They are still lying about it, the issues are not only affecting the console but also AWS operations such as S3 puts. S3 still shows green.
IAM is a "global" service for AWS, where "global" means "it lives in us-east-1".
STS at least has recently started supporting regional endpoints, but most things involving users, groups, roles, and authentication are completely dependent on us-east-1.
Re: AWS us-east-1 outage
#825I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
There is an alternative: A true network cloud. This is what Cloudflare will eventually become.
Re: AWS us-east-1 outage
#826Earlier quoted context omitted.
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…
> This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. > Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. That's an overblown argument. Nobody is saying that, but it's clear that businesses that maintain their ow…
> it's clear that businesses that maintain their own infrastructure would've avoided today's AWS' outage.
Sure, that's trivially obvious. But how many other outages would they have had instead because they aren't as experienced at running this sort of infrastructure as AWS is?
You seem to be arguing from the a priori assumption that rolling your own is inherently more stable than renting infra from AWS, without actually providing any justification for that assumption.
You also seem to be under the assumption that any amount of downtime is always unnacceptable, and worth spending large amounts of time and effort to avoid. For a lot of businesses systems going down for a few hours every once in a while just isn't a big deal, and is much more preferable than spending thousands more on cloud bills, or hiring more full time staff to ensure X 9s of uptime.
Re: AWS us-east-1 outage
#827I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
We have or had alternatives - rackspace, linode, digital ocean, in the past there were many others, self hosting is still an option. But the big three just do it better. The alternatives are doomed to fail. If you use anything other than the big three you risk not just more outages, but your whole provider going out of business overnight.
If the companies at the scale you are talking about do not have multi-region and multi service (aws to azure for example) failover that's their fault, and nobody else's.
Re: AWS us-east-1 outage
#828Earlier quoted context omitted.
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…
Apple created their own silicon. Fedex uses its own pilots. The USPS uses it's own cars. If you're a company relying upon AWS for your business, is it okay if you're down for a day, or two while you wait for AWS to resolve it's issue?
>Apple created their own silicon.
Apple designs the M1. But TSMC (and possibly Samsung) actually manufacture the chips.
Re: AWS us-east-1 outage
#829Earlier quoted context omitted.
Apple created their own silicon. Fedex uses its own pilots. The USPS uses it's own cars. If you're a company relying upon AWS for your business, is it okay if you're down for a day, or two while you wait for AWS to resolve it's issue?
It’s bloody annoying when all I want to do is vacuum the floor and Roomba says nope, “active AWS incident”.
If all you wanted to do was vacuum the floor you would not have gotten that particular vacuum cleaner.
Clearly you wanted to do more than just vacuum the floor and something like this happening should be weighed with the purchase of the vacuum.
Re: AWS us-east-1 outage
#830I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
> Why hasn't the industry come up with an alternative?
We used to have that, some companies still have the capability and know-how to build and run infrastructure that is reliable, distributed across many hosting providers before "cloud" became the "norm", but it goes along with "use or lose it".