Citi Missed a Fat Finger

Probably. AI and LLM are pretty good at detecting extreme outliers and staying on the highly probable path.

Yes, the solution to this problem has to be to shove AI into Citi's order management system somehow.

I always wondered: Since Alarm Fatigue is well known, why do UX designers and/or product owners keep insisting on adding new alarms for things? Surely we know they increasingly don't work, the more you have.

> However, as data from that external feed was unavailable, the price of the value of the Index instead defaulted to -1

To me, this reeks of a couple of bad experiences I had repeatedly as a back end developer coordinating with the front end:

1. Demanding that my services always returned correct-looking data, even if it wasn’t available, because they couldn’t (or wouldn’t) handle errors. The story always went like this: if a service returned a 4xx, it would trigger an exception in their front end code, and they didn’t want to tackle the complexity of handling potential errors from a bunch of different back end services. I’ve had this one get escalated to management before, who fortunately supported me.

2. Resisting efforts to ensure that potentially dangerous values were displayed in attention-getting ways. Anything that didn’t look aesthetically harmonious was liable to get “fixed” without any consideration for why it was presented in a visually jarring way in the first place. As a back end engineer I sometimes lost this battle, since it was outside of my jurisdiction, but when I could, I looped in users or a product manager to help stop the madness.

This triggered a repressed memory. I was once on an ops team responsible for an application that had been turned over to us written in Java. The application ran in Tomcat, and prior to turning it over to us, the external vendor had "helpfully" built some monitoring and alerting for the application. One of the metrics they monitored was how many errors per minute and per hour were happening the application's logs. When they turned this application over to us, they did not properly hand over the monitoring system or even tell us that it existed, and it was sending alerts to another external vendor team which had no association to my ops team.

Several years went by... then because we finally had the time our ops team decided to audit and improve this application, including rewriting significant portions of its code as was pretty typical when we took over an application built by an external vendor. Long story short, we resolved almost all of the errors by the expediency of simply correctly interacting with the database behind the application instead of relying on whatever horrific dumpster fire output to the DB that Hibernate was doing. That's when things got super interesting, because the external vendor team that was receiving these monitoring alerts (unbeknownst to my team) had more than 100% turnover in the intervening period of time, having lost all knowledge about this, but suddenly starting getting a bunch of critical urgency alerts going off because the application was no longer emitting the amount of errors expected for the alarm thresholds. This ended up triggering a massive company-wide incident call, even though there was no external customer impact. I'm leaving a lot of details out, but let's just say it sucked up days of my life and was a massive waste of time for all of that.

Fun example of fixing something and causing an "issue" because behavior changed from expectations/baseline.