Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

151–160 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#151

Post by ashtonkem »

AUX4R6829DR8 wrote: post #61
dijit wrote: post #13

Earlier quoted context omitted.

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

The stuff that's exclusively hosted in us-east-1 is, to my knowledge, mostly things that maintain global uniqueness. CloudFront distributions, Route53, S3 bucket names, IAM roles and similar- i.e. singular control planes. Other than that, regions are about as isolated as it gets, except for specific features on top. Availability zones are supposed to be another fault boundary, and things are generally pretty solid, b…

If I recall there was a point in time where the control panel for all regions was in us-east-1. I seem to recall an outrage where the other regions were up, but you couldn’t change any resources because the management api was down in us-east-1

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#153

Post by fma »

ALittleLight wrote: post #48
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

We have something similar at my big corp company. I think the issue is you went from Green to Red in a flip of a switch. A more normal project goes Green...raise a red flags...if red flags aren't resolved in the next week or two, go to yellow...In these meetings everyone collaborates ways to keep your green or get you back to green if you went yellow.

In essence - what you were saying is your boss lied the whole time, because how does one go from a presumed positive velocity to negative velocity in a week?

Additionally assuming you're a dev lead, it's a little surprising that this is your first meeting of this sorts. As dev lead, I didn't always attend them but my input is always sought on the status.

Sounds like you had a bad manager, and Amazon is filled with them.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#154

Post by Spivak »

betaby wrote: post #152

Problem is that I have to defend our own infrastructure real availability numbers vs cloud's fictional "five nines". It's a loosing game.

All I’m hearing is that you can make up your own availability numbers and get away with it. When you define what it means to be up or down then reality is whatever you say it is.

#gatekeep your real availability metrics

#gaslight your customers with increased error rates

#girlboss

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#155

Post by bdd »

shepherdjerred wrote: post #31
nayuki wrote: post #22

> Operators instead relied on logs to understand what was happening and initially identified elevated internal DNS errors. Because internal DNS is foundational for all services and this traffic was believed to be contributing to the congestion, the teams focused on moving the internal DNS traffic away from the congested network paths. At 9:28 AM PST, the team completed this work and DNS resolution errors fully recove…

It’s quite a bit different… Facebook took themselves offline completely because of a bad BGP update, whereas AWS had network congestion due to a scaling event. DNS relies on the network, so of course it’ll be impacting if networking is also impacted.

no. it wasn't a "bad bgp update". bgp withdrawal of anycast addresses was a desired outcome of a region (serving location) getting disconnected from the backbone. if you'd like to trivialize it, you can say it was configuration change to the software defined backbone.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#156

Post by gkop »

electroly wrote: post #81
logical_proof wrote: post #56

Earlier quoted context omitted.

That is not true. There was hours before they started annotating any kind of service issues. Maybe from when you noticed there was a problem it appeared to be quick, but the board remained green for a large portion of the outtage.

No, it was about an hour. We were aware from the very moment EC2 API error rates began to elevate, around 10:30 Eastern. By 11:30 the dashboard was updating. This timing is mentioned in the article, and it all happened in the middle of our workday on the east coast. The outage then continued for about 7 hours with SHD updates. I suspect we actually both agree on how long it took them to start updating, but I conclude…

At the large platform company where I work, our policy is if the customer reported the issue before our internal monitoring caught it, we have failed. Give 5 minutes for alerting lag, 10 minutes to evaluate the magnitude of impact, 10 minutes to craft the content and get it approved, 5 minutes to execute the update, adds up to 30 minutes end to end with healthy buffer at each step.

1 hour (52 minutes according to the article) sounds meh. I wonder what their error rate and latency graphs look like from that day.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#157

Post by nijave »

jiggawatts wrote: post #91
anonu wrote: post #82

Earlier quoted context omitted.

Wasn't this the Bezos directive early on that created AWS? Anything that was created had to be a service with an API. Not allowed to recreate the wheel. So AWS depends on AWS.

Dependency loops are such fun! My favourite is when some company migrates their physical servers to virtual machines, including the AD domain controllers. Then the next step is to use AD LDAP authentication for the VM management software. When there's a temporary outage and the VMs don't start up as expected, the admins can't log on and troubleshoot the platform because the logon system was running on it... but isn't…

Or even just have some non-federated/"local" accounts stored in a vault somewhere you can use when the centralized auth isn't working

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#158

Post by eyelidlessness »

cyounkins wrote: post #143
foobiekr wrote: post #136

Earlier quoted context omitted.

thundering herd and accidental synchronization for the win I am sad to say, I find issues like this any time I look at retry logic written by anyone I have not interacted with previously on the topic. It is shockingly common even in companies where networking is their bread and butter.

It absolutely is difficult. A challenge I have seen is when retries are stacked and callers time out subprocesses that are doing retries. I just find it amusing that they describe their back-off behaviors as "well tested" and in the same sentence, say it didn't back off adequately.

> It absolutely is difficult. A challenge I have seen is when retries are stacked and callers time out subprocesses that are doing retries.

This is also a general problem with (presumed stateless) concurrent/distributed systems which irked me working on such a system and still haven’t found meaningful resources for which aren’t extremely platform/stack/implementation specific:

A concurrent system has some global/network-wide/partitioned-subset-wide error or backoff condition. If that system is actually stateless and receives push work, communicating that state to them either means pushing the state management back to a less concurrent orchestrator to reprioritize (introducing a huge bottleneck/single or fragile point of failure) or accepting a lot of failed work will be processed in pathological ways.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#159

Post by cookie_monsta »

tw04 wrote: post #97
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Multiple AWS employees have acknowledged it takes VP approval to change the status color of the dashboard. That is absurd and it tells you everything you need to know. The status page isn't about accurate information, it's about plausible deniability and keeping AWS out of the news cycle.

I am so naive. I honestly thought those things were automated.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#160

Post by shagie »

Fordec wrote: post #63
_jal wrote: post #20

Earlier quoted context omitted.

You are clearly not involved in patching.

Simply already patched. Company sizes and number of attack surfaces vary. 22 hours is plenty of time for an input string filter on a centrally controlled endpoint and a dependency increment with the right CI pipeline.

Consider the possible ways for a string to be injected into any of the following:

  Apache Solr
  Apache Druid
  Apache Flink
  ElasticSearch
  Flume
  Apache Dubbo
  Logstash
  Kafka
If you've got any of them, they're likely exploitable too.

That list comes from: https://unit42.paloaltonetworks.com/apache-log4j-vulnerabili...

The attack surface is quite a bit larger than many realize. I recently had a conversation with a person who wasn't at a Java shop so wasn't worried... until he said "oh, wait, ElasticSearch is vulnerable too?"

You'll even see it in things like the connector between CouchBase and ElasticSearch ( https://forums.couchbase.com/t/ann-elasticsearch-connector-4... ).

Post reply on HN