Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

321–330 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#321

Post by raffraffraff »

raffraffraff wrote: post #300
jetru wrote: post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

Interesting. Just wondering if your guys have a dedicated DBA?

Not sure why I got down voted for an honest question. Most start-ups are founders, developers, sales and marketing. Dedicated infrastructure, network and database specialists don't get factored in because "smart CS graduates can figure that stuff out". I've worked at companies who held onto that false notion way too long and almost lost everything as a result ("company extinction event", like losing a lot of customer data)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#322

Post by juanani »

jetru wrote: post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

Bashing trillion dollar behemoths is the right thing to do, by default they spend a certain % on influencing your subconscious brain to get up and defend them everytime someone has a bad article on them. They've probably apent billions on writing positive articles about themselves. They only ask for more and more money, so you get antay when they f up and people are not pleased with them? They aren't jumping up to lick their boots? Fascist shills, can you fuck off to Mars already?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#323

Post by sciurus »

EnlightenedBro wrote: post #254
azundo wrote: post #145

> This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues o…

But what's a good alternative then? What if the internet connection has recovered? And you were at the, for example, 4 minute retry loop. Would you just make your users stare at a spinning loader for 8 minutes?

See for instance the client request rejection probability equation at https://sre.google/sre-book/handling-overload/

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#324

Post by tuldia »

ranguna wrote: post #312
simonbarker87 wrote: post #296

Earlier quoted context omitted.

I’m not all that angry over the situation but more disappointed that we’ve all collectively handed the keys over to AWS because “servers are hard”. Yeh they are but it’s not like locking ourselves into one vendor with flaky docs and a black box of bugs is any better, at least when your own servers go down it’s on you and you don’t take out half of North America.

You can either pay a dedicated team to manage your on prem solution, go multi cloud, or simply go multi region on aws. My company was not affected by this outage because we are multi region. Cheapest and quickest option if you want to have at least some fault tolerance.

> ... multi region. Cheapest and quickest option if you want to have at least some fault tolerance.

That is simple not true, you have to adapt your application to be multi region aware to start with, and if you do that on AWS you are basically locked-in, and one of the most expensive cloud providers out there.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#325

Post by fckyourcloud »

JCM9 wrote: post #55

Queue the armchair infrastructure engineers. The reality is that there’s a handful of people in the world that can operate systems at this sheer scale and complexity and I have mad respect for those in that camp.

There are very few people who can juggle running chainsaws with their penis.

So maybe it's not something we should be doing then?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#326

Post by jmiserez »

gfodor wrote: post #266
colechristensen wrote: post #237

Earlier quoted context omitted.

> System dynamics are hard. And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.

Why is this hard, and can’t just be written down somewhere as part of the engineering discipline? This aspect of systems in 2021 really shouldn’t be an “art.”

It absolutely is written down. The issue is that the results you get from modeling systems using queuing theory are often unintuitive and surprising. On top of that it's hard to account for all the seemingly minor implementation details in a real system.

During my studies we had a course where we built a distributed system and had to model it's performance mathematically. It was really hard to get the model to match the reality and vice-versa. So many details are hidden in a library, framework or network adapter somewhere (e.g buffers or things like packet fragmentation).

We used the book "The Art of Computer Systems Performance Analysis" (R. Jain), but I don't recommend it. At least not the 1st edition which had a frustrating amount of serious, experiment-ruining errata.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#328

Post by WC3w6pXxgGd »

> Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event.

Sentences like this are confusing. If they are well-tested, wouldn't this issue have been covered?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#329

Post by marcinzm »

DenisM wrote: post #49

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.

DAX, part of DynamoDB from how AWS groups things, was throwing internal server errors for us and eventually we had to reboot nodes manually. That's separate from the STS issues we had in terms of our EKS services connecting to DAX.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#330

Post by kortilla »

amzn-throw wrote: post #285
fastball wrote: post #279

Earlier quoted context omitted.

I think most of the outrage is not because "it happened" but because AWS is saying things like "S3 was unaffected" when the anecdotal experience of many in this thread suggests the opposite. That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.

> a VP must sign off on changing status pages, which is... backwards to say the least. I think most people's experience with "VP's" makes them not realize what AWS VP's do. VP's here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to "Call me when the data center is back up and running again!" They are on the tech call, working with the engineers, evaluating…

It doesn’t matter what the VPs are doing, that misses the point. Every minute you know there is a problem and you haven’t at least put up a “degraded” status, you’re lying to your customers.

It was on the top of HN for an hour before anything changed, and then it was still downplayed, which is insane.

Post reply on HN