Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

91–100 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#91

Post by jiggawatts »

anonu wrote: post #82
codegeek wrote: post #79

Earlier quoted context omitted.

"Our Support Contact Center also relies on the internal AWS network, so the ability to create support cases was impacted from 7:33 AM until 2:25 PM PST. " This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here. Also, i…

Wasn't this the Bezos directive early on that created AWS? Anything that was created had to be a service with an API. Not allowed to recreate the wheel. So AWS depends on AWS.

Dependency loops are such fun!

My favourite is when some company migrates their physical servers to virtual machines, including the AD domain controllers. Then the next step is to use AD LDAP authentication for the VM management software.

When there's a temporary outage and the VMs don't start up as expected, the admins can't log on and troubleshoot the platform because the logon system was running on it... but isn't now.

The loop is closed.

You see this all the time, especially with system-management software. They become dependent on the systems they're managing, and vice-versa.

If you care about availability at all, make sure to have physical servers providing basic services like DNS, NTP, LDAP, RADIUS, etc...

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#92

Post by soheil »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

This was addressed at least 3 times during this post. I'm not defending them but you're just gaslighting. If you have something to add about the points they raised regarding the status page please do so.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#93

Post by dastbe »

discodave wrote: post #52
propter_hoc wrote: post #27

Earlier quoted context omitted.

I agree with you, but my services are actually in Canada (Central). There's only one region in Canada, so I don't really have an alternative. AWS justifies it by saying there are three AZs (distinct data centres) within Canada (Central), but I get scared when I see these region-wide issues. If the AZs were really distinct, you wouldn't really have region-wide issues.

Take DynamoDB as an example. The AWS managed service takes care of replicating everything to multiple AZs for you, that's great! You're very unlikely to lose your data. But, the DynamoDB team is running a mostly-regional service. If they push bad code or fall over it's likely going to be a regional issue. Probably only the storage nodes are truly zonal. If you wanted to deploy something similar, like Cassandra across…

> But, the DynamoDB team is running a mostly-regional service.

this is both more and less true than you might think. for most regional endpoints teams leverage load balancers that are scoped zonally, such that ip0 will point at instances in zone a, ip1 will point at instances in zone b, and so on. Similarly, teams who operate "regional" endpoints will generally deploy "zonal" environments, such that in the event of a bad code deploy they can fail away that zone for customers.

that being said, these mitigations still don't stop regional poison pills or otherwise from infecting other AZs unless the service is architected to zonally internally.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#94

Post by electroly »

cyounkins wrote: post #73
nemothekid wrote: post #50

Earlier quoted context omitted.

I've been naively setting up our distributed databases in separate AZs for a couple years now, paying, sometimes, thousands of dollars per month in data replication bandwidth egress fees. As far as I can remember I've never never seen an AZ go down, and the only region that has gone down has been us-east-1.

Is that separate AZs within the same region, or AZs across regions? I didn't think there were any bandwidth fees between AZs in the same region.

It's $0.01/GB for cross-AZ transfer within a region.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#95

Post by bamboozled »

a45a33s wrote: post #41
bamboozled wrote: post #25

Still doesn’t explain the cause of all the IAM permission denied requests we saw against policies which are again working fine without any intervention. Obviously networking issues can cause any number of symptoms but it seems like an unusual detail to leave out to me. Unless it was another ongoing outage happening at the same time.

how are auth requests supposed to reach the auth server if the networking is broken?

I’d accept this as an answer if I received a timeout or a message to say that.

Permission denied is something altogether because it implies the request reached an authorisation system, was evaluated and denied.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#96

Post by multipassnetwrk »

notimetorelax wrote: post #40
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

If you were deployed in 2 regions would it alleviate the impact?

Depends. If your failover to another region required changing DNS and your DNS was using Route 53, you would have problems.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#97

Post by tw04 »

amalter wrote: post #19
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Multiple AWS employees have acknowledged it takes VP approval to change the status color of the dashboard. That is absurd and it tells you everything you need to know. The status page isn't about accurate information, it's about plausible deniability and keeping AWS out of the news cycle.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#98

Post by discodave »

amalter wrote: post #19
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

The AWS summary says: "As the impact to services during this event all stemmed from a single root cause, we opted to provide updates via a global banner on the Service Health Dashboard, which we have since learned makes it difficult for some customers to find information about this issue"

This seems like bad faith to me based on my experience when I worked for AWS. As they repeated many times at Re:Invent last week, they've been doing this for 15+ years. I distinctly remember seeing banners like "Don't update the dashboard without approval from " on various service team runbooks. They tried not to say it out loud, but there was very much a top-down mandate for service teams to make the dashboard "look green" by:

1. Actually improving availability (this one is fair).

2. Using the "Green-I" icon rather than the blue, orange, or red icons whenever possible.

3. They built out the "Personal Health Dashboard" so they can post about many issues in there, without having to acknowledge it publicly.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#100

Post by nemothekid »

electroly wrote: post #94
cyounkins wrote: post #73

Earlier quoted context omitted.

Is that separate AZs within the same region, or AZs across regions? I didn't think there were any bandwidth fees between AZs in the same region.

It's $0.01/GB for cross-AZ transfer within a region.

In reality it's more like $0.02/GB. You pay $0.01 on sending and $0.01 on receiving. I have no idea why ingress isn't free.
Post reply on HN