"The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers." https://health.aws.amazon.com/health/status?path=service-his...
Ah, it's just what I thought. An underlying internal subsystem.
AWS multiple services outage in us-east-1
821–830 of 1001 posts
Re: AWS multiple services outage in us-east-1
#822Do events like this stir conversations in small to medium size businesses to escape the cloud?
It would have to be catastrophic for most businesses to make think about escaping the cloud. The cost of migration and maintenance are massive for small and medium businesses.
Re: AWS multiple services outage in us-east-1
#823I don't think blaming AWS is fair, since they typically exceed their regional and AZ SLAs
AWS makes their SLAs & uptime rates very clear, along with explicit warnings about building failover / business continuity.
Most of the questions on the AWS CSA exam are related to resiliency .
Look, we've all gone the lazy route and done this before. As usual, the problem exists between the keyboard and the chair.
Re: AWS multiple services outage in us-east-1
#824Do events like this stir conversations in small to medium size businesses to escape the cloud?
Depends how small.
I have clients and I’ve heard “even Amazon is down, we can be down” more than once.
Re: AWS multiple services outage in us-east-1
#825Do events like this stir conversations in small to medium size businesses to escape the cloud?
This isn't a "cloud failure". All of these apps would be running now had they spent the additional 5% development costs to add failover to another region.
Re: AWS multiple services outage in us-east-1
#826This is having a direct impact on my wellbeing. I was at Whole Foods in Hudson Yards NYC and I couldn’t get the prime discount on my chocolate bar because the system isn’t working. Decided not to get the chocolate bar. Now my chocolate levels are way too low.
"alexa turn on coffee pot" stopped working this morning, and I'm going bonkers.
Re: AWS multiple services outage in us-east-1
#827Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!
Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at some point. Get your refund and carry on. It's less work for the same outcome.
Re: AWS multiple services outage in us-east-1
#828This is just a silly anecdote, but every time a cloud provider blips, I'm reminded. The worst architecture I've ever encountered was a system that was distributed across AWS, Azure, and GCP. Whenever any one of them had a problem, the system went down. It also cost 3x more than it should.
I've seen the exact same thing at multiple companies. The teams were always so proud of themselves for being "multi-cloud" and managers rewarded them for their nonsense. They also got constant kudos for their heroic firefighting whenever the system went down, which it did constantly. Watching actually good engineers get overlooked because their systems were rock-solid while those characters got all the praise for des…
> Watching actually good engineers get overlooked because their systems were rock-solid while those characters got all the praise for designing an unadulterated piece of shit
That is the computing business. There is no actual accountability, just ass covering
Re: AWS multiple services outage in us-east-1
#829I don't think blaming AWS is fair, since they typically exceed their regional and AZ SLAs AWS makes their SLAs & uptime rates very clear, along with explicit warnings about building failover / business continuity. Most of the questions on the AWS CSA exam are related to resiliency . Look, we've all gone the lazy route and done this before. As usual, the problem exists between the keyboard and the chair.
Not sure any of their SLA’s are covered here.
If they don’t obfuscate the downtime (they will, of course), this outage would put them at, what, two nines? Thats very much out of their SLA.
People also keep talking about it as if its one region, but there are reports in this thread of internal dependencies inside AWS which are affecting unrelated regions with various services. (r53 updates for example)
Re: AWS multiple services outage in us-east-1
#830Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!
> The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud.
If an internal "AWS team" then this translates to "I am comfortable using this tool, and am uninterested in having to learn an entirely new stack."
If you have to diversify your cloud workloads give your devops team more money to do so.