Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

281–290 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#281

Post by electroly »

daguava wrote: post #214
wjossey wrote: post #206

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

You've written up my thoughts better than I can express them myself - I think what people get really stuck on when something like this happens is the 'can I solve this myself?' aspect. A wait for X provider to fix it for you situation is infinitely more stressful than an 'I have played myself, I will now take action' situation. Situations out of your (immediate) resolution control feel infinitely worse, even if the c…

I couldn't possibly disagree more strongly with this. I used to drive frantically to the office to work on servers in emergency situations, and if our small team couldn't solve it, there was nobody else to help us. The weight of the outage was entirely on our shoulders. Now I relax and refresh a status page.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#282

Post by doctor_eval »

dijit wrote: post #21
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Once is a mistake. Twice is a coincidence. Three times is a pattern. But this… This is every time.

Four times is a policy.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#283

Post by jetru »

fastball wrote: post #279
jetru wrote: post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

I think most of the outrage is not because "it happened" but because AWS is saying things like "S3 was unaffected" when the anecdotal experience of many in this thread suggests the opposite. That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.

There's definitely miscommunication around this. I know I've miscommunicated impact, or my communication was misinterpreted across the 2 or 3 people it had to jump before hitting the status page.

For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one convey this in a large org? You run S3, but not STS, it's not technically an S3 fault, but an integration fault across multiple services. If you say S3 is down, you're implying that the storage layer is down. But it's actually not. What's the best answer to make everyone happy here? I cant think of one.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#285

Post by amzn-throw »

fastball wrote: post #279
jetru wrote: post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

I think most of the outrage is not because "it happened" but because AWS is saying things like "S3 was unaffected" when the anecdotal experience of many in this thread suggests the opposite. That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.

> a VP must sign off on changing status pages, which is... backwards to say the least.

I think most people's experience with "VP's" makes them not realize what AWS VP's do.

VP's here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to "Call me when the data center is back up and running again!"

They are on the tech call, working with the engineers, evaluating the problem, gathering the customer impact, and attempting to balance communicating too early with being precise.

Is there room for improvement? Yes. I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message.

But the reason why we don't, doesn't have anything to do with having to get VP approval to put that message up. The VP's are there in the trenches most of the time.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#286

Post by mickeyp »

jetru wrote: post #283
fastball wrote: post #279

Earlier quoted context omitted.

I think most of the outrage is not because "it happened" but because AWS is saying things like "S3 was unaffected" when the anecdotal experience of many in this thread suggests the opposite. That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.

There's definitely miscommunication around this. I know I've miscommunicated impact, or my communication was misinterpreted across the 2 or 3 people it had to jump before hitting the status page. For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one co…

> I cant think of one.

I can.

"S3 is unavailable because X, Y, and Z services are unavailable."

A graph of dependencies between services is surely known to AWS; if not, they ought to create one post-haste.

Trying to externalize Amazon's internal AWS politicking over which service is down is unproductive to the customers who check the dashboard and see that their service ought to be up, but... well, it isn't?

Because those same customers have to explain to their clients and bosses why their systems are malfunctioning, yet it "shows green" on a dashboard somewhere that almost never shows red.

(And I can levy this complaint against Azure too, by the way.)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#287

Post by jetru »

mickeyp wrote: post #286
jetru wrote: post #283

Earlier quoted context omitted.

There's definitely miscommunication around this. I know I've miscommunicated impact, or my communication was misinterpreted across the 2 or 3 people it had to jump before hitting the status page. For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one co…

> I cant think of one. I can. "S3 is unavailable because X, Y, and Z services are unavailable." A graph of dependencies between services is surely known to AWS; if not, they ought to create one post-haste. Trying to externalize Amazon's internal AWS politicking over which service is down is unproductive to the customers who check the dashboard and see that their service ought to be up, but... well, it isn't? Because…

This is a good idea

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#288

Post by Corrado »

mickeyp wrote: post #286
jetru wrote: post #283

Earlier quoted context omitted.

There's definitely miscommunication around this. I know I've miscommunicated impact, or my communication was misinterpreted across the 2 or 3 people it had to jump before hitting the status page. For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one co…

> I cant think of one. I can. "S3 is unavailable because X, Y, and Z services are unavailable." A graph of dependencies between services is surely known to AWS; if not, they ought to create one post-haste. Trying to externalize Amazon's internal AWS politicking over which service is down is unproductive to the customers who check the dashboard and see that their service ought to be up, but... well, it isn't? Because…

Yes, I can envision a (simplified) AWS X-Ray dashboard showing the relationships between the systems and the performance of each one. Then we could see at a glance what was going on. Almost anything is better than that wall of text, tiny status images, and RSS feeds.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#289

Post by gingerlime »

amzn-throw wrote: post #285
fastball wrote: post #279

Earlier quoted context omitted.

I think most of the outrage is not because "it happened" but because AWS is saying things like "S3 was unaffected" when the anecdotal experience of many in this thread suggests the opposite. That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.

> a VP must sign off on changing status pages, which is... backwards to say the least. I think most people's experience with "VP's" makes them not realize what AWS VP's do. VP's here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to "Call me when the data center is back up and running again!" They are on the tech call, working with the engineers, evaluating…

> I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message.

I think that's the crux of the matter? AWS seems to now have a reputation for ignoring issues that are easily observable by customers, and by the time any update shows up, it's way too late. Whether VPs make this decision or not is irrelevant. If this becomes a known pattern (and I think it has), then the system is broken.

disclaimer: I have very little skin in this game. We use S3 for some static assets, and with layers of caching on top, I think we are rarely affected by outages. I'm still curious to observe major cloud outages and how they are handled, and the HN reaction from people on both side of the fence.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#290

Post by xamde »

jetru wrote: post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

There is this really nice website which explains how complex systems fail: https://how.complexsystems.fail/
Post reply on HN