Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

201–210 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#201

Post by 46Bit »

luhn wrote: post #67
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this. They "own up to it" in their postmortems,…

Firmly agreed. I've heard AWS discuss making the status page better – but they get really quiet about actually doing it. In my experience the best/only way to check for problems is to search Twitter for your AWS region name.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#202

Post by virtue3 »

jessermeyer wrote: post #182
foobiekr wrote: post #137

Earlier quoted context omitted.

reality most of the real world successful projects are mostly untested because that's not actually a high ROI endeavor. it kills me to realize that mediocre code you can hack all over to do unnatural things is generally higher value in phase I than the same code done well in twice the time.

This attitude is why modern software is a continuing controlled flight into terrain.

I think the pendulum swing back is going to be designing code that is harder to make bad.

Typescript is a good example of trying to fix this. Rust is even better.

Deno, I think, takes things in a better direction as well.

Ultimately we're going to need systems that just don't let you do "unnatural" things but still maintain a great deal of forward mobility. I don't think that's an unreasonable ask of the future.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#203

Post by soheil »

ripper1138 wrote: post #117
soheil wrote: post #89

Having an internal network like this that everything on the main AWS network so heavily depends on is just bad design. One does not create a stable high tech spacecraft and then fuels it with coal.

Might seem that way because of what happened, but the main network is probably more likely to fail than the internal network. In those cases, running monitoring on a separate network is critical. EC2 control plane same story.

The entire value proposition for AWS is "migrate your internal network to us so it's more stable with less management." I buy that 100%, and I think you're wrong to assume their main network is more likely to fail than their internal one. They have every incentive to continuously improve it because it's not made just for one client.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#204

Post by soheil »

discodave wrote: post #109
soheil wrote: post #89

Having an internal network like this that everything on the main AWS network so heavily depends on is just bad design. One does not create a stable high tech spacecraft and then fuels it with coal.

It's 2006, you work for an 'online book store' that's experimenting with this cloud thing. Are you going to build a whole new network involving multi-million dollar networking appliances?

No but one would hope 15 years and 1 trillion dollars later you would stop running it on the computer under your desk.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#205

Post by geenew »

blackearl wrote: post #164
koheripbal wrote: post #161

Earlier quoted context omitted.

This challenge is not specific to Amazon. Being able to automatically detect system health is a non-trivial effort.

Why automatic? Surely someone could have the responsibility to do it manually.

Or override the autogenerated values

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#206

Post by wjossey »

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise.

First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region.

Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up, keep it as generally available as is reasonable.

Third, you’re gonna go down when the cloud goes down. Not much use getting overly bent out of shape. You can reduce your exposure by just using their core systems (EC2, S3, SQS, LBs, Cloudfrount, RDS, Elasticache). The more systems you use, the less reliable things will be. However, running your own key value store, api gateway, event bud, etc., can also be way less reliable than using their’s. So, realize it’s an operational trade off.

Degradation of your app / platform is more likely to come from you than AWS. You’re gonna roll out bad code, break your own infra, overload your own system, way more often than Amazon is gonna go down. If reliability matters to you, start by examining your own practices first before thinking things like multi region or super durable highly replicated systems.

This stuff is hard. It’s hard for Amazon engineers. Hard for platform folks at small and mega companies. It’s just, hard. When your app goes down, and so does Disney plus, take some solace that Disney in all their buckets of cash also couldn’t avoid the issue.

And, finally, hold cloud providers accountable. If they’re unstable and not providing service you expect, leave. We’ve got tons of great options these days, especially if you don’t care about proprietary solutions.

Good luck y’all!

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#207

Post by mnordhoff »

banana_giraffe wrote: post #83
propter_hoc wrote: post #7

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

It can be a bit hard to know, since the AZ identifiers are randomized per account, so if you think you have problems in us-west-1a, I can't check on my side. You can get the AZ ID out of your account to de-randomize things, so we can compare notes, but people rarely bother, for whatever reason.

Amazon seems to have stopped randomizing them in newer regions. Another reason to move to us-east-2. ;-)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#208

Post by davidstoker »

DenisM wrote: post #49

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.

Do you use VPC endpoints for S3? The next sentence explained failures I observed with S3: "However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event."

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#209

Post by DenisM »

davidstoker wrote: post #208
DenisM wrote: post #49

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.

Do you use VPC endpoints for S3? The next sentence explained failures I observed with S3: "However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event."

No, I use the normal endpoint.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#210

Post by redconfetti »

subsaharancoder wrote: post #142
ALittleLight wrote: post #48

Earlier quoted context omitted.

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

I worked at AMZN and this perfectly captures my experience there with those weekly reviews. I once set a project I was managing as "Red" and had multiple SDMs excoriate me for apparently "throwing them under the bus" even though we had missed multiple timelines and were essentially not going to deliver anything of quality on time. I don't miss this aspect of AMZN!

How dare you communicate a problem using the color system. It hurts feelings, and feelings are important here.
Post reply on HN