I am going to take a public bet this was an issue caused by a code deployment to the AWS service Amazon Event Bridge...
AWS us-east-1 outage
961–970 of 1001 posts
Re: AWS us-east-1 outage
#962Earlier quoted context omitted.
Can someone just have a site ping all the GET endpoints on the AWS API? That is very far from "automating [their entire] system" but it's better than what they're doing.
Something like this? https://stop.lying.cloud/
Um, that's just a reskin of AWS's lying status page. No real new data there.
Re: AWS us-east-1 outage
#963Earlier quoted context omitted.
depends I guess, I am running on-prem workstation for our DWH. So far in 2 years it went down minutes at the time, when I decided to do so, because of hardware updates. I have no idea where this narrative came from, but usually hardware you have is very reliable and doesn't turn off every 15 minutes. Heck, I use old T430 for my home server and still it doesn't go down on completely random occasions (but thats very si…
But was it always accessible from the internet, and serving requests in an acceptable amount of time?
The one in work yes, but for internal network, as we are not exposed to internet. But to be honest, we are probably one of few companies that make priority that there is always electricity and internet in the office (with UPS, electricity generator, multiple internet providers).
No idea what are the standards for other companies.
Re: AWS us-east-1 outage
#964FYI, there was a Redshift query outage this morning, and Redshift COPY operations from S3 to Redshift have been failing for quite a while now. Status page shows all green but AWS confirms differently when on a customer phone call.
Redshift Concurrency Scaling seems like it went down and UNLOAD to S3 operations were also failing much of the day. Seems like all has been working fine the last 3-4 hours.
Re: AWS us-east-1 outage
#965I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…
> you own your dependencies. You don't blame others.
I love that. Build your service to be robust. Never assume that dependencies are 100% reliable. Gracefully handle failures. Don't just go hard down, or worse, sure horribly in a way that you can't recover from automatically when you're dependencies come back. I've seen a single database outage cause cascading failures across a whole site even though most services had no direct connection to the database. And recovery had to be done in order of dependency, or else you're playing whack-a-mole for an hour)
> VP approval is needed to make updates on the status board.
Isn't that normal? Updating the status has a cost (reparations to customers if you breach SLA). You don't want some on-call engineer stressing over the status page while trying to recover stuff.
Re: AWS us-east-1 outage
#966The most annoying aspect of this for us was that the AWS Organizations API only has a single endpoint, in us-east-1.
https://docs.aws.amazon.com/general/latest/gr/ao.html
As a result, even if you weren’t using that region but you were using the API you were hosed for 6+ hours, And the status page never acknowledged that it was out of action.
We had the following Terraform in our production pipeline:
data “aws_organizations_organization” “current” {}
As a result, all of our deployments to our EU regions were borked. Of course, we couldn’t raise a support case because the support system was also down, and despite escalating to our TAM weren’t able to get the status page to reflect reality.
My concern is that the Organizations API specifically will be brushed under the carpet and we will still have a single point of failure in a region which we never intend to use.
Re: AWS us-east-1 outage
#967Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…
> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…
> Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region?
They're cheap. HA is for their customers to pay more, not for Amazon which often lies during major outages. They would lose money on HA and they would lose money on acknowledging downtimes. They will lie as long as they benefit from it.
Re: AWS us-east-1 outage
#968Earlier quoted context omitted.
> Apple created their own silicon Apple designed their own silicon, a third party manufactures and packages it for them.
Pedantic, -1.
I suggest you review this before commenting again:
Re: AWS us-east-1 outage
#969Earlier quoted context omitted.
I'll play devil's advocate here and say that sometimes these incidents deserve praise because they uncovered an issue that was otherwise unknown previously. Also if the incident had a large negative impact then it shows to leadership how critical normal operation of that service is. Even if you were the cause of the issue, the fact that you fixed it and kept the critical service operating the rest of the time, is wor…
I know; that's not what I'm asking about. I'm talking about a different issue.
Mistakes happen, and a culture that insists too hard that "mistakes shouldn't happen, and so we can't be seen making mistakes" is harmful toward engineering.
How should their performance be evaluated, if not by the rote number of mistakes that can be pinned onto the person, and their combined impact? (Was that the question?)
Re: AWS us-east-1 outage
#970Earlier quoted context omitted.
And that is why my lighting automation has a baseline req that it works 100% without the internet and preferably without a central controller.
So basically it's just a toggle switch?
My lighting automation uses Insteon currently. My primary req is that they are smart and connected without needing a central controller o a connection to the internet. My switches all understand lighting scenes and can manage those in a P2P manner, without a central controller. The central controller is primarily used when I want to add actual automations vs. scenes. Even the central controller aspect works 100% disconnected from the internet though. I can easily layer on top any automations I like. For instance, I have my exterior lights driven my the angle of the sun. Then, on top of that I can add internet based triggers for automation as needed. This is where I add in voice assistant triggering of automation and scenes.
Edit: Just to add, very simple binary automations are even possible without a central controller. Like, I have Insteon motion sensors that trigger a lighting scene when they detect motion. These are super simplistic though.