51–60 of 152 posts
us-east-1 continues to have continually worse uptime than other regions (for, likely, good reason too, it continues to be the default region). I've avoided that region and I can't remember the last time I had downtime caused by Amazon.
Also, it is one of the regions that gets new features first, which makes me wonder if it contributes to lower stability.
https://status.heroku.com/incidents/1892 - it appears Heroku is being particularly affected. We've had multiple sites on multiple accounts go down in the past few minutes. EDIT T16:31Z: It appears Heroku has failed over their dashboard, but dynos are still failing to come online. We had assumed that they had multi-region failovers for their customers. Incredibly disappointing.
Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.
Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system. An outage like this happens how often? Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're…
Without data it doesn't even cost significantly more than running in a single region nowadays, if you are willing to go serverless. As serverless stuff (FaaS, ...) is pay for what you use and the provider handles the scaling automatically behind the curtain you can easily deploy to multiple regions without much additional cost.
With data you have of course the cost of storing the data multiple times in the different regions (or to come up with some kind of sharding) and solving the consistency challenges that come with that, but at least services like DynamoDB and S3 offer cross-region replication out of the box nowadays and you don't have to provision any capacity like you used to (thanks to DynamoDB AutoScaling and so on).
Once you have your application running in multiple regions you can direct users to the closest one, so they enjoy lower latencies.
I believe for a lot of applications running cross-region just makes a lot of sense as it offers various benefits.
I'm in Australia and Reddit/Twitter ground to a standstill - request timeout after request timeout. I presumed it was an outage somewhere but was surprised to learn it was with AWS us-east-1? I would have thought surely that my connection would have referenced a different region based on my location.
Designing truly resilient and available applications with DB servers that replicate across continents is hard.
I’m surprised by how much of the “internet” seem to be affected by a single AZ going down.
Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.
Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system. An outage like this happens how often? Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're…
My engineers are all React and CSS web developers. They don't know anything about multi tenant data resiliency. But they can make a real pretty "system down" page.
https://status.heroku.com/incidents/1892 - it appears Heroku is being particularly affected. We've had multiple sites on multiple accounts go down in the past few minutes. EDIT T16:31Z: It appears Heroku has failed over their dashboard, but dynos are still failing to come online. We had assumed that they had multi-region failovers for their customers. Incredibly disappointing.
We cannot even restart/turn off the dynos or get into the dashboard to turn off and kill our background tasks for some of our clients.
It looks like it was localized to zone D.
Zone designations are account specific; zone D for you is not zone D for me
I'm in Australia and Reddit/Twitter ground to a standstill - request timeout after request timeout. I presumed it was an outage somewhere but was surprised to learn it was with AWS us-east-1? I would have thought surely that my connection would have referenced a different region based on my location.
usually, DB servers will live in a small number of locations with good connectivity between clusters and the frontends (which terminate the user's TCP connection) live much closer (likely Sydney). Good design means that there are few roundtrips between the FEs and the backend but they are not unavoidable. Designing truly resilient and available applications with DB servers that replicate across continents is hard.
I guess partitioning can help, but then isn't it just turning the DB servers into pizzas of master-slave where the Hawaiian slice is master only in Hawaii, and slave everywhere else?