Why does that webpage renders like a dog?... I get that it's under load but the rendering itself is chugging something rare.
Edit: wow that webpage is humongous... never heard of paging?
AWS us-east-1 outage
771–780 of 1001 posts
Re: AWS us-east-1 outage
#772I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…
[deleted]
Re: AWS us-east-1 outage
#773The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…
If you did that some clever person would set up their PiHole so that their device just always worked, and then you couldn't send them ads and surveil them. They'd tell their friends and then everyone would just use their local devices locally. Totally irresponsible what you're suggesting.
this is why everyone runs piholes and no one sees ads on the internet anymore, which killed the internet ad industry
Re: AWS us-east-1 outage
#774Re: AWS us-east-1 outage
#775I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
In my opinion there is a lack of talent in these industries for building out there own resilient systems. IT people and engineers get lazy.
We're too busy in endless sprints to focus on things outside of our core business that don't make salespeople and executives excited.
Re: AWS us-east-1 outage
#776I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…
We knew us-east-1 was unuseable for our customers for 45 minutes before amazon acknowledged anything was wrong _at all_. We made decisions _in the dark_ to serve our customers, because amazon drug their feet communicating with us. Our customers were notified after 2 minutes.
It's not acceptable.
Re: AWS us-east-1 outage
#777I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
Because the expected value of using AWS is greater than the expected value of self-hosting. It's not that nobody's ever heard of running on their own metal. Look back at what everyone did before AWS, and how fast they ran screaming away from it as soon as they could. Once you didn't have to do that any more, it's just so much better that the rare outages are worth it for the vast majority of startups. Medical devices…
Agree with your first point.
On the second though, at some point, infrastructure like AWS are going to be more reliable than what many banks, medical device operators etc can provide themselves. asking them to stay on their own hardware is asking for that industry to remain slow, bespoke and expensive.
Re: AWS us-east-1 outage
#778I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…
> In-house servers would lead to an insane amount of outage.
That might be true, but the effects of any given outage would be felt much less widely. If Disney has an outage, I can just find a movie on Netflix to watch instead. But now if one provider goes down, it can take down everything. To me, the problem isn't the cloud per se, it's one player's dominance in the space. We've taken the inherently distributed structure of the internet and re-centralized it, losing some robustness along the way.
Re: AWS us-east-1 outage
#779I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…
Apple created their own silicon. Fedex uses its own pilots. The USPS uses it's own cars.
If you're a company relying upon AWS for your business, is it okay if you're down for a day, or two while you wait for AWS to resolve it's issue?
Re: AWS us-east-1 outage
#780Any leads on the cause? Some services on us-east-1 seem to be working just fine. Others not.