I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.
Not right now. I think they monitor if it appears on HN too.
AWS us-east-1 outage
331–340 of 1001 posts
Re: AWS us-east-1 outage
#332Re: AWS us-east-1 outage
#333Earlier quoted context omitted.
Would they? Having 3 outages in a year sounds like an organization problem. Not enough safeguards to prevent very routine human errors. But instead of worrying about that we just assign a guy to take the fall
You're not wrong, but it's possible that the organization is small enough that it's just not feasible to have enough safeguards that would prevent the outages John caused. And in that case, it's probably best that John not be promoted if he can't avoid those errors.
Current co is small. We are putting in the safeguards from Day 1. Well, okay technically like day 120, the first few months were a mad dash to MVP. But now that we have some breathing room, yeah, we put a lot of emphasis on preventing outages, detecting and diagnosing outages promptly, documenting them, doing the whole 5-why's thing, and preventing them in the future. We didn't have to, we could have kept mad dashing and growth hacking. But very fortunately, we have a great culture here (founders have lots of hindsight from past startups).
It's like a seed for crystal growth. Small company is exactly the best time to implement these things, because other employees will try to match the cultural norms and habits.
Re: AWS us-east-1 outage
#334I'm now getting failures searching for products on Amazon.com itself. This is somewhat surprising, as the narrative always was that Amazon didn't do a great job of dogfooding their own cloud platform.
Doesn't amazon.com run on us-east-1?
Re: AWS us-east-1 outage
#335This is effecting heroku. While my heroku apps are currently up, I am unable to push new versions. Logging in to heroku dashboard (which does work), there is a message pointing to this heroku status incident for "Availability issues with upstream provider in the US region": https://status.heroku.com/incidents/2390 How can there be an outage severe enough to be effecting middleman customers like heroku, but the AWS st…
AWS management APIs in the us-east-1 region is what is affected. I'm guessing Heroku uses at least the S3 APIs when deploying new versions, and those are failing (intermittently?).
I advise not touching your Heroku setup right now. Even something like trying to restart a dyno might mean it doesn't come back since the slug is probably stored on S3 and that will fail.
Re: AWS us-east-1 outage
#336I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…
I worked at Walmart Technology. I bravely wrote post mortem documents owning the fault of my team (100+ people), owning both technically and also culturally as their leader. I put together a plan to fix it and executed it. Thought that was the right thing to do. This happend two times in my 10 year career there.
Both times I was called out as a failure in my performance eval. Second time, I resigned and told them to find a better leader.
Happy now I am out of such shitty place.
Re: AWS us-east-1 outage
#337Earlier quoted context omitted.
I have no inside knowledge or anything but it seems like there are a lot of scenarios with degraded performance where people could argue about whether it really constitutes an outage.
One time gcp argued that since they did return 404s on gcs for a few hours that wasn’t an uptime/latency sla violation so we were not entitled to refund (tho they refunded us anyway)
Man, between costs and shenanigans like this, why don't more companies self-host?
Re: AWS us-east-1 outage
#338Need google voice unlimited,
Re: AWS us-east-1 outage
#339...and imdb
Re: AWS us-east-1 outage
#340I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
And this is exactly why you can expect these headlines to hit with great regularity. These things are never a problem at the individual level, they are always at the level of culture and organization.