Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

151–160 of 214 posts

Re: AWS North Virginia data center outage – resolved

#151

Post by roryirvine »

somat wrote: post #131
dlenski wrote: post #113

Earlier quoted context omitted.

Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on a…

We had a pair of machines. And some bright spark set them up to mount each others NFS shares. after a power outage "Holy mother of chicken and egg NFS hangs batman" That was a weird job, fun, it was a local machine room for a warehouse that originally held the IBM mainframe, it still held it's successor "the multiprise 3000" which has the claim to fame as being the smallest mainframe IBM ever sold. But now the room w…

Oh, yeah, re-exporting NFS mounts via SMB was very much a thing in the early 2000s - something to do with their different approaches to flock() vs fcntl() handling. If you ran into locking issues with nfs, then re-exporting via SMB was standard advice.

At some point, the behaviour changed and locks starting conflicting. IIRC, we hit it when upgrading to Debian Etch and took the time to unwind the system and make pure NFS work properly for us. Plenty of people took the opposite approach, and fiddled with the config to make locking a noop on SMB. I know of at least one web hosting company who ended up having to restore a year's worth of customer uploads from backups as a result...

Re: AWS North Virginia data center outage – resolved

#152

Post by Roark66 »

dlenski wrote: post #101
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Anyone who thinks one cloud provider will provide them full resilience is fooling themselves. You need multicloud for true high availability.

But then you want to use the same stack across providers and all the proprietary technologies (even hidden from you with things like terraform) are suddenly loosing their luster.

Re: AWS North Virginia data center outage – resolved

#153

Post by michaelt »

MichaelZuo wrote: post #116
dlenski wrote: post #113

Earlier quoted context omitted.

Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on a…

Why would such a critical server even be accessible with only one set of keys? I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.

They're not actually accessible with 'only one set of keys' in my experience.

You actually have to present your photo ID at the site entry gatehouse, then again to the building entry guard (who will also check you have a work permit and a site-specific safety induction) then you swipe a badge at a turnstile to get from reception into the stairwell, then swipe your badge at a door to get into the relevant floor, then swipe your badge and key in a code to enter the room with the cages then you use the key.

Re: AWS North Virginia data center outage – resolved

#154

Post by Zopieux »

zaphirplane wrote: post #62
shimman wrote: post #43

Earlier quoted context omitted.

It's a good thing big tech hires for ethical engineers and not ones that only care about money or social status.

You forgot the /s

What's funny about sarcasm is that you're supposed to detect it without markers.

Re: AWS North Virginia data center outage – resolved

#155

Post by teitoklien »

altern8 wrote: post #138
tornikeo wrote: post #133

I wonder if hetzner had better uptime in EU than AWS this year.

Why no love for OVH? I find Hetzner's UI to be super-confusing, making it hard to manage things.

ovh rocks, they are also far more customer support friendly than hetzner. Half the time hetzner feels like they are doing you a favor by letting you rent servers from them.

Ovh is way simpler, and openstack integration from them works good enough for most of my needs.

Kinda insane how atrocious docs are tho. No .md markdown format to let agents read stuff yet -_-

Re: AWS North Virginia data center outage – resolved

#157

Post by cmiles8 »

zaphirplane wrote: post #148
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Services outside of us-east-1 don’t call us-east-1 for IAM data plane thou right ?

They’re talking about the backbone and what goes on behind the scenes. There have been issues with services in other regions when us-east-1 has issues.

Folks built in other regions believing they were fully isolated only to discover later during an outage that they were not.

Re: AWS North Virginia data center outage – resolved

#159

Post by myroon5 »

dlenski wrote: post #101
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

> outside of China

[Nitpick] There are a few more AWS partitions like GovCloud:

https://jasonbutz.info/2023/07/aws-partitions/

Re: AWS North Virginia data center outage – resolved

#160

Post by noosphr »

aurareturn wrote: post #37

These things are dangerous. Someone who can take AWS down such as an employee can place a bet. These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.

Jokes on you. All betting sites are based on US-East1.
Post reply on HN