AWS us-east-1 outage

I'm not saying you're wrong, or that the status page is architected properly. I'm just speaking to the current situation.

This is effecting heroku.

While my heroku apps are currently up, I am unable to push new versions.

Logging in to heroku dashboard (which does work), there is a message pointing to this heroku status incident for "Availability issues with upstream provider in the US region": https://status.heroku.com/incidents/2390

How can there be an outage severe enough to be effecting middleman customers like heroku, but the AWS status page is still all green?!?!

If whoever runs the AWS status page isn't embaressed, they really ought to be.

I am finding that I have a very bimodal response to "He did it". When I write an RCA or just talk about near misses, I may give you enough details to figure out that Tom was the one who broke it, but I'm not going to say Tom on the record anywhere, with one extremely obvious exception.

If I think Tom has a toxic combination of poor judgement, Dunning-Kruger syndrome, and a hint of narcissism (I'm not sure but I may be repeating myself here), such that he won't listen to reason and he actively steers others into bad situations (and especially if he then disappears when shit hits the fan), then I will nail him to a fucking cross every chance I get. Public shaming is only a tool for getting people to discount advice from a bad actor. If it comes down to a vote between my idea and his, then I'm going to make sure everyone knows that his bets keep biting us in the ass. This guy kinda sounds like the Toxic Tom.

What is important when I turned out to be the cause of the issue is a bit like some court cases. Would a reasonable person in this situation have come to the same conclusion I did? If so, then I'm just the person who lost the lottery. Either way, fixing it for me might fix it for other people. Sometimes the answer is, "I was trying to juggle three things at once and a ball got dropped." If the process dictated those three things then the process is wrong, or the tooling is wrong. If someone was asking me questions we should think about being more pro-active about deflecting them to someone else or asking them to come back in a half hour. Or maybe I shouldn't be trying to watch training videos while babysitting a deployment to production.

If you never say "my bad" then your advice starts to sound like a lecture, and people avoid lectures so then you never get the whole story. Also as an engineer you should know that owning a mistake early on lets you get to what most of us consider the interesting bit of solving the problem instead of talking about feelings for an hour and then using whatever is left of your brain afterward to fix the problem. In fact in some cases you can shut down someone who is about to start a rant (which is funny as hell because they look like their head is about to pop like a balloon when you say, "yep, I broke it, let's move on to how do we fix it?")

In my experience this is the actual reason for fear of the formal error correction process.

Looks like Kinesis Firehose is either the root cause, or severely impacted:

https://twitter.com/amontalenti/status/1468265799458639877

Segment is publicly reporting issues delivering to Firehose, and one of my company's real-time monitors also triggered for Kinesis Firehose an hour ago.

Update:

By my sniff of it, some “core” APIs are down for S3 and EC2 (e.g. GET/PUT on S3 and node create/delete on EC2). Systems like Kinesis Firehose and DynamoDB rely on these APIs under the hood (“serverless” is just “a server in someone else’s data center”).

Further update:

There is a workaround available for the AWS Console login issue. You can use https://us-west-2.console.aws.amazon.com/ to get in -- it's just the landing page that is down (because the landing page is in the affected region).