Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

301–310 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#301

Post by nix23 »

simonbarker87 wrote: post #296
jetru wrote: post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

I’m not all that angry over the situation but more disappointed that we’ve all collectively handed the keys over to AWS because “servers are hard”. Yeh they are but it’s not like locking ourselves into one vendor with flaky docs and a black box of bugs is any better, at least when your own servers go down it’s on you and you don’t take out half of North America.

Servers are not hard if you have a dedicated person (long time ago known as Systemadminstrator), and fun fact...it's sometimes even much cheaper and more reliable then having everything in the "cloud".

Personally i am a believer in mixed environments, public webservers etc in the "cloud", locally used systems and backup "in house" with a second location (both in Data-centers or at least one), and no, i don't talk about the next google but the 99% of businesses.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#302

Post by darkwater »

speedgoose wrote: post #251
bombcar wrote: post #249

Earlier quoted context omitted.

True multi-cloud redundancy is hard to test - because it’s everything from DNS on up and it’s hard to ask AWS to go offline so you can verify Azure picks up the slack.

It's true, but you can do load balancing at the DNS level.

And you will get 1/N of requests timing or erroring out, and in the meanwhile paying 2x or 3x the costs. So, it might be worth in some cases but you need to evaluate it very, very well.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#303

Post by [deleted] »

herodoturtle wrote: post #297

I am grateful to AWS for this report. Not sure if any AWS support staff are monitoring this thread, but the article said: > Customers also experienced login failures to the AWS Console in the impacted region during the event. All our AWS instances / resources are in EU/UK availability zones, and yet we couldn't access our console either. Thankfully none of our instances were affected by the outage, but our inability…

[deleted]

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#304

Post by daenney »

divbzero wrote: post #256

> This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it. Disruption of the standard incident response mechanism seems to be a common element of longer lasting incidents.

Yup. There was a GCP outage a couple of years ago like this.

I don’t remember the exact details, but it was something along the lines of a config change went out that caused systems to incorrectly assume there were huge bandwidth constraints. Load shedding kicked in to drop lower priority traffic which ironically included monitoring data rendering GCP responders blind and causing StackDriver to go blank for customers.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#305

Post by 0x0nyandesu »

gingerlime wrote: post #289
amzn-throw wrote: post #285

Earlier quoted context omitted.

> a VP must sign off on changing status pages, which is... backwards to say the least. I think most people's experience with "VP's" makes them not realize what AWS VP's do. VP's here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to "Call me when the data center is back up and running again!" They are on the tech call, working with the engineers, evaluating…

> I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message. I think that's the crux of the matter? AWS seems to now have a reputation for ignoring issues that are easily observable by customers, and by the time any update shows up, it's way too late. Whether VPs make this decision or not is irrelevant. If this becomes a known pattern (and I think it has), then…

Saying "S3 is down" can mean anything. Our S3 buckets that served static web content stayed up no problem. The API was down though. But for the purposes of whether my organization cares I'm gonna say it was "up".

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#306

Post by shakna »

0x0nyandesu wrote: post #305
gingerlime wrote: post #289

Earlier quoted context omitted.

> I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message. I think that's the crux of the matter? AWS seems to now have a reputation for ignoring issues that are easily observable by customers, and by the time any update shows up, it's way too late. Whether VPs make this decision or not is irrelevant. If this becomes a known pattern (and I think it has), then…

Saying "S3 is down" can mean anything. Our S3 buckets that served static web content stayed up no problem. The API was down though. But for the purposes of whether my organization cares I'm gonna say it was "up".

> We are currently experiencing some problems related to FOO service and are investigating.

A generic, utterly meaningless message, which is still a hell of a lot more than usually gets approved, and approved far too late.

It is also still better than "all green here, nothing to see" which has people looking at their own code, because they _expect_ that they will be the problem, not AWS.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#307

Post by dclowd9901 »

gfodor wrote: post #266
colechristensen wrote: post #237

Earlier quoted context omitted.

> System dynamics are hard. And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.

Why is this hard, and can’t just be written down somewhere as part of the engineering discipline? This aspect of systems in 2021 really shouldn’t be an “art.”

Think of other extremely complex systems and how we’ve managed to make them stable:

1) airplanes: they crashed, _a lot_. We used data recorders and stringent process to make air travel safety commonplace.

2) cars: so many accidents accident research. The solution comes after the disaster.

3) large buildings and structures: again, the master work of time, attempts, failures, research and solutions.

If we really want to get serious about this (and I think we do) we need to stop reinventing infrastructure every 10 years and start doubling down on stability. Cloud computing, in earnest, has only been around a short while. I’m not even convinced it’s the right path forward, just happens to align best with business interests, but it seems to be the devil we’re stuck with so now we need to really dig in and make it solid. I think we’re actually in that process right now.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#308

Post by brentcetinich »

jetru wrote: post #283
fastball wrote: post #279

Earlier quoted context omitted.

I think most of the outrage is not because "it happened" but because AWS is saying things like "S3 was unaffected" when the anecdotal experience of many in this thread suggests the opposite. That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.

There's definitely miscommunication around this. I know I've miscommunicated impact, or my communication was misinterpreted across the 2 or 3 people it had to jump before hitting the status page. For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one co…

It doesn’t matter about the dependency graph , but on the definition of unavailable for s3 in its sla

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#309

Post by raffraffraff »

Something they didn't mention is AWS Billing alarms. These rely on metrics systems which were affected by this (and are missing some data). Crucially, billing alarms only exist in the us-east-1 region, so if you're using them, your impacted no matter where you're infrastructure is deployed. (That's just my reading of it)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#310

Post by ikiris »

randmeerkat wrote: post #36
dijit wrote: post #13

Earlier quoted context omitted.

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

AWS has been getting a pass on their stability issues in us-east-1 for years now because it’s their “oldest” zone. Maybe they should invest in fixing it instead of inventing new services to sell.

if you care about the availability of a single geographical availability zone, it's your own fault.
Post reply on HN