It's always East 1... Jokes aside I don't understand how often east-1 is taken down compared to other regions. Like it should be pretty similar to other regions architecture wise.
AWS North Virginia data center outage – resolved
51–60 of 214 posts
Re: AWS North Virginia data center outage – resolved
#52Earlier quoted context omitted.
You have a back up for the back up backup. Turtles all the way down. At AWS scale even unlikely hardware events become more common I guess.
Each turtle gives them another 9. How many 9s are they down due to incidents over the past year?
They're definitely more than half a day now, which is only two and a half nines.
Re: AWS North Virginia data center outage – resolved
#53Earlier quoted context omitted.
Lots of proposals to build them near Lake Michigan recently but the residents of Wisconsin only want auto parts stores and paper mills. They've been completely demonized. Cities and counties are passing no data center laws even though it's the perfect place for it.
Paper mills need a lot of heat energy to run the processes. Data centres produce a lot of heat. Sounds like a good combination? Cold water -> data centre cooling loop - > warm water -> paper mill with heat pumps to transform low-grade heat into the required temperatures -> profit
Data center cooling system output has miserably low extractable energy.
Re: AWS North Virginia data center outage – resolved
#54Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat
Off the top of my head: Ocean levels of salt in a water system are much more expensive to maintain (even the secondary loop). Coastal land much more expensive. If you go to a remote coastal site, you probably won't have as good access to power. Coastal sites usually exposed to more severe weather events. Other fun unpredicatble things eg-Diablo Canyon nuclear facility has had issues with debris and jellyfish migratio…
And oysters / mussels / clams / every other creature that starts small and turns calcium into brick finds your cooling system to be a delightful place to raise a family, especially in delicate heat exchangers with small easily blockable passages.
Re: AWS North Virginia data center outage – resolved
#55Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat
Humidity and corrosion, it's a trade-off (pick your poison).
Re: AWS North Virginia data center outage – resolved
#56Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat
So Ashburn VA is a datacenter hub because the very first non-government Internet Exchange Point (IXP) anywhere in the world was there ( https://en.wikipedia.org/wiki/MAE-East ). Back in the 1990's something like half of all internet traffic all over the world hit MAE-East. That in turn made AWS put their first region there (us-east-1 preceded eu-west-1 by 2 years and us-west-1 by 3 years). Then because there were lot…
The underlying reason is more that by being in us east coast you have about equal latency for customers in us west coast and Europe. That's a very large population covered from a single site.
If you're building a single datacenter site this is where you start building first.
Re: AWS North Virginia data center outage – resolved
#57AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.
anecdotally (well, more "second-hand-ly i heard that..." it sounds like there were some carry-on effects on us-east-2 as a result of people migrating over from us-east-1, so, yeah... kinda hilarious how the multiple region / AZ thing is just so plainly a façade, but yet we all seem to just collectively believe in it as an article of faith in the Cloud Religion... or whatever...
It's no magic given the size of us-east-1 there is no spare capacity to absorb all the workloads
Re: AWS North Virginia data center outage – resolved
#58It's always East 1... Jokes aside I don't understand how often east-1 is taken down compared to other regions. Like it should be pretty similar to other regions architecture wise.
Isn't east one the "core" datacenter and also the oldest? I'd imagine it has more load than the other regions and also has more tech debt and architectural / engineering debt because they had less experience when they built it.
Also iirc some services rely on east-1 as a single point of failure for configuration (like IAM or some S3 stuff?)
Re: AWS North Virginia data center outage – resolved
#59How many nines of are we at this year?
Re: AWS North Virginia data center outage – resolved
#60AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.
Is it really failing more, or we just don’t hear about failure happening elsewhere?
Last i heard azure outage it wasn’t even on HN frontpage