301–310 of 1001 posts
Friends tell friends to pick us-east-2. Virginia is for lovers, Ohio is for availability.
I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…
People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat model of decay. They don't do it adding new dependencies (build- or runtime), and therefore are unprepared to handle an outage.
There's a good reason for this, of course: modern software "best practices" encourage moving fast and breaking things, which includes "add this dependency we know nothing about, and which gives an unknown entity the power to poison our code or take down our service, arbitrarily, at runtime, but hey its a cool thing with lots of github stars and it's only one 'npm install' away".
Just want to end with this PSA: Dependencies bad.
I think its just the console - as my EC2 in us-east-1 are still reachable.
Azure, Google Cloud, AWS and others need to have a “Status alliance” where they determine the status of each of their services by a quorum using all cloud providers. Status pages are virtually useless these days
I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
What if they just can't access the console to update the status page...
Earlier quoted context omitted.
I don't think engineers can believe in no-blame analysis if they know it'll harm career growth. I can't unilaterally promote John Doe, I have to convince other leaders that John would do well the next level up. And in those discussions, they could bring up "but John has caused 3 incidents this year", and honestly, maybe they'd be right.
Would they? Having 3 outages in a year sounds like an organization problem. Not enough safeguards to prevent very routine human errors. But instead of worrying about that we just assign a guy to take the fall
We make heavy usage of Kinesis Firehose in us-east-1. Issues started ~1:24am ET and resolved around 7:31am ET. Then really kicked in at a much larger scale at 10:32am ET. We're now seeing failures with connections to RDS Postgres and other services. Console is completely unavailable to me.
I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
I've had push backs on my postmortems before because of phrasing that could be constituted as laying some of the blame on some person/team when it's supposed to be blameless.
And for a long time, it was fairly blameless. You would still be punished with the extra work of writing high quality postmortems, but I have seen people accidentally bring down critical tier-1 services and not be adversely affected in terms of promotion, etc.
But somewhere along the way, it became politicized. Things like the wheel of death, public grilling of teams on why they didn't follow one of the thousands of best practices, etc, etc. Some orgs are still pretty good at keeping it blameless at the individual level, but... being a big company, your mileage may vary.
I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.
It's like trying to get the truth out of a kid that caused some trouble. Mom: Alexa, did you break something? Alexa: No. M: Really? What's this? 500 Internal server error A: ok maybe management console is down M: Anything else? A: ... A: ... ok maybe cloudwatch logs M: Ah hah. What else? A: That's it, I swear! M: 503 ClientError A: ...well okay secretsmanager might be busted too...
Me: Alexa, is AWS down right now?
Alexa: I'd rather not answer that
Earlier quoted context omitted.
That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…
I don't think engineers can believe in no-blame analysis if they know it'll harm career growth. I can't unilaterally promote John Doe, I have to convince other leaders that John would do well the next level up. And in those discussions, they could bring up "but John has caused 3 incidents this year", and honestly, maybe they'd be right.
-You don't hide anything
-Errors will be made
-After training/mission everyone talks about the errors (or potential ones) and how to prevent them
-You don't make the same error twice
Being afraid to make errors and learn from them creates a culture of hiding, a culture of denial and especially being afraid to take responsibility.