Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

381–390 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#381

Earlier quoted context omitted.

I assumed in good faith that this is someone knowing internals as a larger customer, not an AWS person shit-talking other AWS teams. Got curious only after a downvote hence late edit. My bad.

> ...an AWS person shit-talking other AWS teams [in public]. I remember a time when this would be an instant reprimand... Either amzn engs are bolder these days, or amzn hr is trying really hard for amzn to be "world's best employer", or both.

It's true that I shouldn't have posted it, was mostly just in a grumpy mood. It's still considered very bad form. I'm not actually there anymore, but the idea stands.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#382
post #154

Earlier quoted context omitted.

All I’m hearing is that you can make up your own availability numbers and get away with it. When you define what it means to be up or down then reality is whatever you say it is. #gatekeep your real availability metrics #gaslight your customers with increased error rates #girlboss

What are you trying to imply with that last hashtag?

It’s a meme; search it on Twitter. It’s a play on “live, laugh, love” that started as a way for young women to mock pandering displays of female empowerment but has grown in scope so that it can be used to mock anyone.

     #gatekeep #gaslight #girlboss
or the male equivalent

     #mansplain #manipulate #malewife

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#383

Earlier quoted context omitted.

> Third, you’re gonna go down when the cloud goes down. Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug. The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't…

True multi-cloud redundancy is hard to test - because it’s everything from DNS on up and it’s hard to ask AWS to go offline so you can verify Azure picks up the slack.

I deeply concur with this statement. I think folks here are conflating a one off test versus keeping your redundancy up to date as apps evolve.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#384

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

> Safety of your customer’s data is your ethical responsibility. Protect it, back it up, keep it as generally available as is reasonable. > Third, you’re gonna go down when the cloud goes down. Not much use getting overly bent out of shape. “Whoops, our provider is down, sorry!” is not taking responsibility with customer data at all.

Respectfully disagree. No company in the world has 100% uptime. Whether it’s your server rack or their server rack going down means nothing to a customer.

We’re not discussing data loss in this thread specifically. This is about a couple of hours of downtime per year.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#385

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

> Third, you’re gonna go down when the cloud goes down. Not much use getting overly bent out of shape. Ugh. I have a hard time with this one. Back in the day, EBS had some really awful failures and degradations. Building a greenfield stack that specifically avoided EBS and stayed up when everyone else was down during another mass EBS failure felt marvelous. It was an obvious avoidable hazard. It doesn't mean "avoid E…

I hear you. I didn’t use EBS for five years after the great outage in, what was it, 2011?

At this point, it’s reliable enough that even if it were to go down, it’s more safe than not using it. I’d put EBS in the pantheon of “core” services I never mind using these days.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#386
post #283

Earlier quoted context omitted.

There's definitely miscommunication around this. I know I've miscommunicated impact, or my communication was misinterpreted across the 2 or 3 people it had to jump before hitting the status page. For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one co…

> I cant think of one. I can. "S3 is unavailable because X, Y, and Z services are unavailable." A graph of dependencies between services is surely known to AWS; if not, they ought to create one post-haste. Trying to externalize Amazon's internal AWS politicking over which service is down is unproductive to the customers who check the dashboard and see that their service ought to be up, but... well, it isn't? Because…

Later on in the process, you could do something like this. When you know what else is impacted and how that looks to your customers. But by then the problem is most likely over or at least on the way to being fixed. And hours may have gone by before you get to that point.

Early in the process, when you’re flying blind because you don’t know what’s going on around you and you look at your own systems and they appear to be fine, you can’t really say anything useful.

These weird edge cases are hard to adjudicate because they’ve never happened before — otherwise fixes would already be in place to prevent them. And nothing quite like them has ever before happened at this scale.

I understand the frustration, but when everything you think you know turns out to be wrong, or at least you are unable to confirm whether it’s right or wrong, what do you do?

Read the RCA — When AWS got to that point, they did actually update the SHD with a banner across the top of the page, but that ended up actually causing even more problems. There’s a reason why you try to do these sorts of things safely, which may mean using manual methods in some cases. And sometimes even those safe manual methods have their own weird side effects.

Sometime shit is hard. Sometimes you run into problems like no one else on the planet has ever experienced before, and you have to figure out what the laws of physics are in this new part of the world as you go about trying to fix whatever it was that broke or acted in an unexpected manner.

Disclaimer: my opinions are my own and are not necessarily shared or reflective of my employer.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#387
post #293

I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…

> And, finally, hold cloud providers accountable. If they’re unstable and not providing service you expect, leave. We’ve got tons of great options these days, especially if you don’t care about proprietary solutions. Easy to say, but difficult to do in practice (leaving a cloud provider)

Absolutely hard. But that doesn’t mean if you’re in a position to start a company from scratch that you can’t walk away. Or if you go to another company and are involved in their procurement of a new purchase, that you can’t sway it away from said provider.

Just because it takes years doesn’t meant it can’t happen.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#388
post #301

Earlier quoted context omitted.

I’m not all that angry over the situation but more disappointed that we’ve all collectively handed the keys over to AWS because “servers are hard”. Yeh they are but it’s not like locking ourselves into one vendor with flaky docs and a black box of bugs is any better, at least when your own servers go down it’s on you and you don’t take out half of North America.

Servers are not hard if you have a dedicated person (long time ago known as Systemadminstrator), and fun fact...it's sometimes even much cheaper and more reliable then having everything in the "cloud". Personally i am a believer in mixed environments, public webservers etc in the "cloud", locally used systems and backup "in house" with a second location (both in Data-centers or at least one), and no, i don't talk abo…

You could have a staff of a million-plus people and stuff could still go sideways.

Hint — it did.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#389
post #353

Earlier quoted context omitted.

Not one person, at least four people to run stuff 24/7.

99% of businesses don't need 24/7 but two are the bare minimum (a admin, and a dev or admin)

That’s like saying that you’ve got a hard drive so you could be your own DropBox.

You vastly underestimate the amount of resources required. And what those resources cost.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#390
post #278

Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem. We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and…

But Amazon advertises that they DO understand the complexity of this, and that their understanding, knowledge and experience is so deep that they are a safe place to put your critical applications, and so you should pay them lots of money to do so. Totally understand that complex systems behave in incomprehensible ways (hopefully only temporarily incomprehensible). But they're selling people on the idea of trading yo…

They don’t sell “guaranteed no downtime throughout the history of the universe”.

They have SLAs. And there are clauses that cover the weird edge cases for when the SLAs are not met.

Post reply on HN