Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

181–190 of 214 posts

Re: AWS North Virginia data center outage – resolved

#181

Post by nilamo »

Eridrus wrote: post #98
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

> And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

Is it not a selling point to be able to say "we're still up while out competitors are down"?

Re: AWS North Virginia data center outage – resolved

#182

Post by martinald »

strgcmc wrote: post #175
cagenut wrote: post #170

Earlier quoted context omitted.

there are dozens of us!

There are often little bits of Neal Stephenson or Andy Weir novels which sound a little like this, describing a technical fault in a plot-driven way (often as a cascade), and I do find those to be uniquely enjoyable. I'm sure there are other authors who do similar things, though maybe "cloud/AI data center" stories should be its own micro-genre, given how crucial these things are to society.

I wrote this recently which maybe people will enjoy in the same vein :) https://martinalderson.com/posts/august-29-2026-a-scenario/

Re: AWS North Virginia data center outage – resolved

#183

Post by grogenaut »

aurareturn wrote: post #37

These things are dangerous. Someone who can take AWS down such as an employee can place a bet. These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.

Oh no won't someone please think of the prop wagerers.

Re: AWS North Virginia data center outage – resolved

#186

Post by noAnswer »

teitoklien wrote: post #155
altern8 wrote: post #138

Earlier quoted context omitted.

Why no love for OVH? I find Hetzner's UI to be super-confusing, making it hard to manage things.

ovh rocks, they are also far more customer support friendly than hetzner. Half the time hetzner feels like they are doing you a favor by letting you rent servers from them. Ovh is way simpler, and openstack integration from them works good enough for most of my needs. Kinda insane how atrocious docs are tho. No .md markdown format to let agents read stuff yet -_-

They once had offerings for dedicated servers without hard drivers. They did network boot from NFS. So the costs where between a full dedicated server and a virtual one. Sadly it was very badly engineered. Small disk IO was so bad that you basically couldn't run MySQL. I did run a MX. For every mail postfix would complained that the filesystem did run a few secondes in the future. At some point they gave up and stuck a USB stick into every server.

It was dead by thousand cuts and put a bad taste in my mouth. But I have to admit that was a long time ago and I should probably give them another chance.

Re: AWS North Virginia data center outage – resolved

#187

Post by easton »

master_crab wrote: post #165
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

IAM isn’t even really the most painful dependency. Route53 is. The control plane only runs out of use1. Better make sure the only DNS operations you run during an outage are data plane queries and health check failovers.

They actually kind of fixed this recently, you can ask them to move your route53 control plane to another region in the event of us-east-1 breaking: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ac...

There’s a bunch of caveats but it’s worth enabling if you’re changing dns all the time (as most AWS networking doodads like to do).

Re: AWS North Virginia data center outage – resolved

#188

Post by y3ahd0g »

cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

No. This is nonsense.

Some SaaS apps had issues.

The Internet was fine.

This is physical reality. The internet was designed to route around this.

Just because some app devs do a lazy job doesn't mean the entire infrastructure as designed is garbage.

Just because some app devs are over reliant on a single cloud service doesn't mean the Internet is broken.

Re: AWS North Virginia data center outage – resolved

#189

Post by martin8412 »

superjan wrote: post #136
sidewndr46 wrote: post #103

Earlier quoted context omitted.

Isn't this kind of circular dependency what lead to extended downtime a while back?

It reminds me of facebook. Staff was locked out of the office due to the outage they were supposed to fix.

Luckily the plasma torch and bolt cutter didn’t require logging in with Facebook.

Re: AWS North Virginia data center outage – resolved

#190

Post by dehrmann »

cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

> building across multiple regions and AZs is a thing

If you do this for resiliency, be prepared to pay the capacity tax (2 regions means 2x capacity, 3 regions means 1.5x), have the machines already running in a multi-region setup (don't expect to be able to spin up instances or even get capacity during an outage), and ready to deal with the added complexity of multi-region hosting.

Post reply on HN