121–130 of 840 posts
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".
Earlier quoted context omitted.
> I see all green checks on the AWS Status page. I'm sure you know this already, but that status page isn't worth the cycles on your CPU, you would be better served asking the toaster if AWS is functioning properly than checking that status page.
Of course, yeah, but at least you can sometimes see a yellow and infer it really means red :/.
edit: it is now showing as a total outage on the status page
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".
These events seem to be happening almost on a monthly basis now. IRC was never this unreliable and at least with netsplits it was obvious what had happened because you'd see the clients disconnect. IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice. It's really poor.
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".
- high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, the higher the divergence world of production and the world of the new feature
- lots of new hires (new year = new hiring budget). New hires are missing some tribal knowledge about the system and make a production-breaking release.
I tried to think of other reasons, but these two overwhelmingly stand out as the two biggest reasons. Would love to hear from others.
Earlier quoted context omitted.
I'm sitting here not sure if I should deploy code since most communication with the rest of my team has been cut off.
If something went awry, and it caused more pain because Slack was down, how would you feel? If you’re missing comms/observability then waiting to deploy seems prudent.
I'm migrating to Rocket.chat on Digital Ocean as we speak. Has anyone else made the move or tested Rocket.chat?
These events seem to be happening almost on a monthly basis now. IRC was never this unreliable and at least with netsplits it was obvious what had happened because you'd see the clients disconnect. IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice. It's really poor.
My feeling is this is an AWS issue. Our services hosted in AWS are not working either.