Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

Post by speedgoose » Sat, Dec 11, 2021, 5:52 AM UTC

bombcar wrote: post #249

speedgoose wrote: post #245

Earlier quoted context omitted.

> Third, you’re gonna go down when the cloud goes down. Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug. The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't…

True multi-cloud redundancy is hard to test - because it’s everything from DNS on up and it’s hard to ask AWS to go offline so you can verify Azure picks up the slack.

Post by vmception » Sat, Dec 11, 2021, 5:55 AM UTC

vmception wrote: post #252

colechristensen wrote: post #237

Earlier quoted context omitted.

> System dynamics are hard. And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.

> Most of them are designs based on nothing but uninformed intuition. Or because they read it on a Google|AWS Engineering blog

Post by EnlightenedBro » Sat, Dec 11, 2021, 6:07 AM UTC

manquer wrote: post #238

wjossey wrote: post #206

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

Easy to say leave, the techinical lockin cloud service providers by design choose to have makes it impossible to leave . AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily. If the advise is to use only effectively use manag…

Post by tybit » Sat, Dec 11, 2021, 6:12 AM UTC

hericium wrote: post #240

WaxProlix wrote: post #231

Earlier quoted context omitted.

STS is the worst with this. Even for other internal teams, they seem to treat dropped requests (ie, timeouts which represent 5xxs on the client side) as 'non faults', and so don't treat those data points in their graphs and alarms. It's really obnoxious. AWS in general is trying hard to do the right thing for customers, and obviously has a long ways to go. But man, a few specific orgs have some frustrating holdover p…

> AWS in general is trying hard to do the right thing for customers You are responding to a comment that suggests they're misrepresenting the truth (which wouldn't be the first time even in last few days) in communication to their customers. As always, they are doing the right thing for themselves only. EDIT: I think that you should mention being an Engineer at Amazon AWS in your comment.

Post by femiagbabiaka » Sat, Dec 11, 2021, 6:13 AM UTC

divbzero wrote: post #256

> This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it. Disruption of the standard incident response mechanism seems to be a common element of longer lasting incidents.