Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

231–240 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#231

Post by WaxProlix »

tyingq wrote: post #174

"Amazon Secure Token Service (STS) experienced elevated latencies" I was getting 503 "service unavailable" from STS during the outage most of the time I tried calling it. I guess by "elevated latency", they mean from anyone with retry logic that would keep trying after many consecutive attempts?

STS is the worst with this. Even for other internal teams, they seem to treat dropped requests (ie, timeouts which represent 5xxs on the client side) as 'non faults', and so don't treat those data points in their graphs and alarms. It's really obnoxious.

AWS in general is trying hard to do the right thing for customers, and obviously has a long ways to go. But man, a few specific orgs have some frustrating holdover policies.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#232

Post by nijave »

markranallo wrote: post #102
JCM9 wrote: post #4

Obviously one hopes these things don’t happen, but that’s an impressive and transparent write up that came out quickly.

Its not transparent at all. A massive amount of services were hard down for hours like SNS and were never acknowledged on the status page or in this write-up. This honestly reads like they don't truly understand the scope of things effected.

It sounded like the entire management plane was down and potentially part of the "data" plane too (management being config and data being get/put/poll to stateful resources)

I saw in the Reddit thread someone mentioned all services that auth to other services on the backend were effected (not sure how truthful it is but that certainly made sense)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#233

Post by nijave »

tuananh wrote: post #135
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

even in the post mortem, they are reclutant to admit it > While AWS customer workloads were not directly impacted from the internal networking issues described above, the networking issues caused impact to a number of AWS Services which in turn impacted customers using these service capabilities. Because the main AWS network was not affected, some customer applications which did not rely on these capabilities only ex…

>some customer applications which did not rely on these capabilities only experienced minimal impact from this event

Yeah so vanilla LB and EC2 with no autoscaling were fine. Anyone using "serverless" or managed services had a real bad day

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#234

Post by notinty »

DrBenCarson wrote: post #125
JCM9 wrote: post #85

Earlier quoted context omitted.

They address that in the post, and between Twitter, HN and other places there wasn’t anyone legit questioning if something was actually broken. Contacts at AWS also all were very clear that yes something was going on and being investigated. This narrative that AWS was pretending nothing was wrong just wasn’t true based on what we saw.

I'm going to leave it at this: the dashboards at AWS aren't automated. Say what you will, but I can automate a status dashboard in a couple days--yes, even at AWS scale. No reason the dashboard should be green for hours while their engineers and support are aware things aren't working.

Apparently VP approval is required to update it, i.e. they're a farce.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#235

Post by manquer »

banana_giraffe wrote: post #87
DenisM wrote: post #49

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.

Yeah > For example, while running EC2 instances were unaffected by this event That ignores the 3-5% drop in traffic I saw in us-east-1 on EC2 instances that only talk to peers on the Internet with TCP/IP during this event.

I guess you have to read this kind of items hidden in careful language, running the instances had no problem, it is different matter they had limited connectivity!from AWS point of view they don't seem to see user impact but services from their point of view.

Perhaps that distinction has value if your workloads did not depend on network connectivity externally for example say S3 access without vpc and only compute some DS/ ML jobs perhaps.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#236

Post by dalyons »

daguava wrote: post #214
wjossey wrote: post #206

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

You've written up my thoughts better than I can express them myself - I think what people get really stuck on when something like this happens is the 'can I solve this myself?' aspect. A wait for X provider to fix it for you situation is infinitely more stressful than an 'I have played myself, I will now take action' situation. Situations out of your (immediate) resolution control feel infinitely worse, even if the c…

For me it’s the opposite… aws outages are much less stressful than my own because I know there’s nothing I/we can do about it, they have smart people working on it, and it will be fixed when it’s fixed

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#237

Post by colechristensen »

azundo wrote: post #145

> This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues o…

> System dynamics are hard.

And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#238

Post by manquer »

wjossey wrote: post #206

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

Easy to say leave, the techinical lockin cloud service providers by design choose to have makes it impossible to leave .

AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily.

If the advise is to use only effectively use managed open source components then why AWS at all ? most competent mid sized teams can do that much cheaper with a colo providers like OVH/hetzner.

The point of investing in AWS is not outsource running base infra, if we should stay away from leveraging the kind of cloud native services us mere mortals cannot hope to build or maintain.

Also this avoid us-east-1 advice is bit frustrating, AWS does not have to experiment with new services always in the same region,it is not marked as experimental region or has reduced SLAs , if it is inferior/preview/beta than call it out in the UI and contract, what about when there is no choice? If cloudfront is managed in us-east-1 and we shouldnt now use it ? Why use the cloud then ?

if your engineering only discovers scale problems at us-east-1 along with customers perhaps something is wrong ? aws could limit new instances in that region and spread the load, playing with customers like this who are at your mercy just because you can is not nice.

Disney can afford to go down, or build their cloud, small companies don't have deep pockets to do either

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#239

Post by oasisbob »

wjossey wrote: post #206

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

> Third, you’re gonna go down when the cloud goes down. Not much use getting overly bent out of shape.

Ugh. I have a hard time with this one. Back in the day, EBS had some really awful failures and degradations. Building a greenfield stack that specifically avoided EBS and stayed up when everyone else was down during another mass EBS failure felt marvelous. It was an obvious avoidable hazard.

It doesn't mean "avoid EBS" is good advice for the decade to follow, but accepting failure fatalistically doesn't feel right either.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#240

Post by hericium »

WaxProlix wrote: post #231
tyingq wrote: post #174

"Amazon Secure Token Service (STS) experienced elevated latencies" I was getting 503 "service unavailable" from STS during the outage most of the time I tried calling it. I guess by "elevated latency", they mean from anyone with retry logic that would keep trying after many consecutive attempts?

STS is the worst with this. Even for other internal teams, they seem to treat dropped requests (ie, timeouts which represent 5xxs on the client side) as 'non faults', and so don't treat those data points in their graphs and alarms. It's really obnoxious. AWS in general is trying hard to do the right thing for customers, and obviously has a long ways to go. But man, a few specific orgs have some frustrating holdover p…

> AWS in general is trying hard to do the right thing for customers

You are responding to a comment that suggests they're misrepresenting the truth (which wouldn't be the first time even in last few days) in communication to their customers.

As always, they are doing the right thing for themselves only.

EDIT: I think that you should mention being an Engineer at Amazon AWS in your comment.

Post reply on HN