Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

131–140 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#131

Post by edoceo »

systemvoltage wrote: post #54
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

People of HN has been extremely unprofessional with regards to AWS's downtime. Some kind of a massive zeitgeist against Amazon, like a giant hive mind that spews hate. Why are we doing this folks? What's making you so angry and contemptful? Literally try searching the history of downtimes and it was always professional and respectful. Yesterday, my comment was fricking flagged for asking people to be nice to which pe…

All too often folk conflate frustration with anger or hate.

The comments are frustrated users.

Not hateful.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#132

Post by sorry_outta_gas »

s_dev wrote: post #32
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

>You can take it on bad faith It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless. If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.

That's only useful when it's an entire region, there are minor issues in smaller services that cause problems for a lot of people they don't reflect in their status board; and not everyone checks twitter or HN all the time while at work

it's a bullshit board used fudge numbers when negoaiting SLAs

like I don't care that much, hell my company does the same thing; but let's not get defensive over it

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#133

Post by zeko1195 »

Fordec wrote: post #63
_jal wrote: post #20

Earlier quoted context omitted.

You are clearly not involved in patching.

Simply already patched. Company sizes and number of attack surfaces vary. 22 hours is plenty of time for an input string filter on a centrally controlled endpoint and a dependency increment with the right CI pipeline.

lol no

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#135

Post by tuananh »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

even in the post mortem, they are reclutant to admit it

> While AWS customer workloads were not directly impacted from the internal networking issues described above, the networking issues caused impact to a number of AWS Services which in turn impacted customers using these service capabilities. Because the main AWS network was not affected, some customer applications which did not rely on these capabilities only experienced minimal impact from this event.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#136

Post by foobiekr »

cyounkins wrote: post #42

My favorite sentence: "Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event."

thundering herd and accidental synchronization for the win

I am sad to say, I find issues like this any time I look at retry logic written by anyone I have not interacted with previously on the topic. It is shockingly common even in companies where networking is their bread and butter.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#137

Post by foobiekr »

DrBenCarson wrote: post #122
discodave wrote: post #103

Earlier quoted context omitted.

I saw pleeeeeenty of untested code at Amazon/AWS. Looking back it was almost like the most important services/code had the least amount of testing. While internal boondoggle projects (I worked on a couple) had complicated test plans and debates about coverage metrics.

This is almost always the case. The most important services get the most attention from leaders who apply the most pressure, especially in the first ~2y of a fast-growing or high-potential product. So people skip tests.

reality most of the real world successful projects are mostly untested because that's not actually a high ROI endeavor. it kills me to realize that mediocre code you can hack all over to do unnatural things is generally higher value in phase I than the same code done well in twice the time.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#139

Post by ClifReeder »

almostdeadguy wrote: post #88

> The AWS container services, including Fargate, ECS and EKS, experienced increased API error rates and latencies during the event. While existing container instances (tasks or pods) continued to operate normally during the event, if a container instance was terminated or experienced a failure, it could not be restarted because of the impact to the EC2 control plane APIs described above. This seems pretty obviously f…

Strong agree. We were using Fargate nodes in our us-east-1 EKS cluster and not all of our nodes dropped, but every coredns pod did. When they came back up their age was hours older than expected, so maybe a problem between Fargate and the scheduler rendered them “up” but unable to be reached?

Either way, was surprising to us that already provisioned compute was impacted.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#140

Post by lanstin »

cyounkins wrote: post #42

My favorite sentence: "Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event."

I found myself wishing for a few code snippets here. It would be interesting. A lot of time code that handles "connection refused" or fast failures doesn't handle network slowness well. I've seen outages from "best effort" services (and the best-effort-ness worked when the services were hard down) because all of a sudden calls that were taking 50 ms were not failing but all taking 1500+ ms. Best effort but no client enforced SLAs that were low enough to matter.

Load shedding never kicked in, so things had to be shutdown for a bit and then restarted.

Seems their normal operating state might be what is called "meta-stable" - dynamically stable at a high thru-put (edited) unless/until a brief glitch bumps the system into the low work being finished state which is also stable.

Post reply on HN