Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

131–140 of 214 posts

Re: AWS North Virginia data center outage – resolved

#131

Earlier quoted context omitted.

It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.

Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on a…

We had a pair of machines. And some bright spark set them up to mount each others NFS shares. after a power outage "Holy mother of chicken and egg NFS hangs batman"

That was a weird job, fun, it was a local machine room for a warehouse that originally held the IBM mainframe, it still held it's successor "the multiprise 3000" which has the claim to fame as being the smallest mainframe IBM ever sold. But now the room was also full of decades of artisanal crafted unix servers with pick databases. the pick dev team had done most the system architecture. The best way to understand it is that for them pick is the operating system, unix is a necessary annoyance they have to put up with only because nobody has made pick hardware for 20 years. and it was NFS mounts everywhere, somebody had figured out a trick where they could NFS mount a remote machine and have the local pick system reach in and scrounge through the remote systems data. But strictly read-only. pick got grumpy when writing to NFS not to say anything about how the other database would feel about having it's data being messed with. Thus the circular mount.

Still was not the worst thing I saw. I liked the one system with a SMB mount. "Why is this one SMB?" "Well pick complains when you try to write to a NFS mount, but it's NFS detection code does not trip on SMB mounts." ... Sighs "Um... I am no pick expert but you know why it does not like remote mounts right. SMB does not change that, Do you happen to get a lot of corrupt indexes on this machine?" "yes, how did you know"

Re: AWS North Virginia data center outage – resolved

#134

Earlier quoted context omitted.

Why would such a critical server even be accessible with only one set of keys? I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.

Other than for certain nuclear missile launches[1], that only happens in the movies. [1] https://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact...

I dont know how it is in the datacentre industry, but certainly in other industries that is how its done for anything truly mission critical and also easily tampered with.

I guess it shows very few care enough to pay enough to make that a reasonable upgrade.

Re: AWS North Virginia data center outage – resolved

#136

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Isn't this kind of circular dependency what lead to extended downtime a while back?

It reminds me of facebook. Staff was locked out of the office due to the outage they were supposed to fix.

Re: AWS North Virginia data center outage – resolved

#137
post #43

These things are dangerous. Someone who can take AWS down such as an employee can place a bet. These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.

It's a good thing big tech hires for ethical engineers and not ones that only care about money or social status.

[flagged]

Re: AWS North Virginia data center outage – resolved

#140

Earlier quoted context omitted.

yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack. It's basically a wash for almost all organizations for twice the cost and effort.

also these things don't go down THAT often... well aws, not some others. More uptime that you probably had before. even the stock market takes a few days off every decade. Just ask W.

> not some others.

Looking at Azure and GitHub in particular. ;)

Post reply on HN