This outage is a reminder:
Economic efficiency and technical complexity are both, separately and together, enemies of resilience
AWS multiple services outage in us-east-1
841–850 of 1001 posts
Re: AWS multiple services outage in us-east-1
#842Can confirm, also getting hit with this.
Re: AWS multiple services outage in us-east-1
#843I realize that my basement servers have better uptime than AWS this year! I think most sysadmin don't plan for AWS outage. And economically it makes sense. But it makes me wonder, is sysadmin a lost art?
That's not very surprising. At this point you could say that your microwave has a better uptime. The complexity comparison to all the Amazon cloud services and infrastructure would be roughly the same.
So, we need to centralise all our compute onto one provider because economies of scale and its too complicated to do it well…
… but we should not compare them to self-hosting because hosting that much data and compute is complicated.
The emperor has no clothes.
Re: AWS multiple services outage in us-east-1
#844That's unusual. I wss under the impression that having multiple available zones guarantees high availability. It seems this is not the case.
One of the open secrets of AWS is that even though AWS has a lot of regions and availability zones, a lot of AWS services have control planes that are dependent on / hosted out of us-east-1 regardless of which region / AZ you're using, meaning even if you are using a different availability zone in a different region, us-east-1 going down still can mess you up.
Re: AWS multiple services outage in us-east-1
#845Cool, building in resilience seems to have worked. Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted (not sure if it would have been anyway). My control plane is native multi-region, so while it depends on many impacted services it stayed available. Each region runs in isolation. There is data replication at play but failing to replicate to us-east-1 had no impact on other…
active/active? curious what the data stack looks like as that tends to be the hard part
The data layer is DynamoDB with Global Tables providing replication between regions, so we can write to any region. It's not easy to get this right, but our use case is narrow enough and rate of change low enought (intentionally) that it works well. That said, it still isn't clear that replication to us-east-1 would be perfect so we did "diff" tables just to be sure (it has been for us).
There is some S3 replication as well in the CI/CD pipeline, but that doesn't impact our customers directly. If we'd seen errors there it would mean manually taking Virginia out of the pipeline so we could deploy everyehere else.
Re: AWS multiple services outage in us-east-1
#846Stupid question, why isn't the stock down? Couldn't this lead to people jumping to other providers and at the very least require some pretty big fees for do dramatically breaking SLA? Is it just not a biggest fraction of revenue to matter?
Robinhood is down ;)
Re: AWS multiple services outage in us-east-1
#847Cool, building in resilience seems to have worked. Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted (not sure if it would have been anyway). My control plane is native multi-region, so while it depends on many impacted services it stayed available. Each region runs in isolation. There is data replication at play but failing to replicate to us-east-1 had no impact on other…
How did you do resilient auth for keys and certs?
We use AWS for keys and certs, with aliases for keys so they resolve properly to the specific resources in each region. For any given HTTP endpoint there is a cert that is part of a the stack in that region (different regions use different certs).
The hardest part is that our customers' resources aren't always available in multiple regions. When they are we fall back to a region where they exist that is next closest (by latency, courtesy of https://www.cloudping.co/).
Re: AWS multiple services outage in us-east-1
#848Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close. Our company decided years ago to use any region other than us-east-1. Of course, that doesn't help with services that are 'global', which usually means us-east-1.
Several reasons, really:
1. The main one: it's the cheapest region, so when people select where to run their services they pick it because "why pay more?"
2. It's the default. Many tutorials and articles online show it in the examples, many deployment and other devops tools use it as a default value.
3. Related to n.2. AI models generate cloud configs and code examples with it unless asked otherwise.
4. It's location make it Europe-friendly, too. If you have a small service and you'd like to capture European and North American audience from a single location us-east-1 is a very good choice.
5. Many Amazon features are available in that region first and then spread out to other locations.
6. It's also a region where other cloud providers and hosting companies offer their services. Often there's space available in a data center not far from AWS-running racks. In hybrid cloud scenarios where you want to connect bits of your infrastructure running on AWS and on some physical hardware by a set of dedicated fiber optic lines us-east-1 is the place to do it.
7. Yes, for AWS deployments it's an experimental location that has higher risks of downtime compared to other regions, but in practice when a sizable part of us-east-1 is down other AWS services across the world tend to go down, too (along with half of the internet). So, is it really that risky to run over there, relatively speaking?
It's the world's default hosting location, and today's outages show it.
Re: AWS multiple services outage in us-east-1
#849Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close. Our company decided years ago to use any region other than us-east-1. Of course, that doesn't help with services that are 'global', which usually means us-east-1.
We have discussions coming up to evict ourselves from AWS entirely. Didn't seem like there was much of an appetite for it before this but now things might have changed. We're still small enough of a company to where the task isn't as daunting as it might otherwise be.
Re: AWS multiple services outage in us-east-1
#850Stupid question, why isn't the stock down? Couldn't this lead to people jumping to other providers and at the very least require some pretty big fees for do dramatically breaking SLA? Is it just not a biggest fraction of revenue to matter?
Non-technical people don't really notice these things. They hear it and shrug, because usually it's fixed within a day.
CNBC is supposed to inform users about this stuff, but they know less than nothing about it. That's why they were the most excited about the "Metaverse" and telling everyone to get on board (with what?) or get left behind.
The market is all about perception of value. That's why Musk can tweet a meme and double a stocks price, it's not based in anything real.