Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Citi Missed a Fat Finger

bloomberg.com

Post reply on HN
Print view View original

21–30 of 92 posts

Re: Citi Missed a Fat Finger

#21

Post by thephyber »

ryandrake wrote: post #15
blackeyeblitzar wrote: post #9

Earlier quoted context omitted.

This is a phenomenon in so many different jobs, and is called “alarm fatigue”. From https://www.nytimes.com/2010/08/01/weekinreview/01wald.html : > On Tuesday, the National Transportation Safety Board said that a crash last year on the Washington subway system that killed nine people had happened partly because train dispatchers had been ignoring 9,000 alarms per week. Air traffic controllers, nuclear plant operators…

I always wondered: Since Alarm Fatigue is well known, why do UX designers and/or product owners keep insisting on adding new alarms for things? Surely we know they increasingly don't work, the more you have.

Alarms are easy for a single person to add.

The better solution is to redesign / engineer systems which automatically solve the alarm condition so no person needs to know about the alarm. That usually requires many people to work together and requires far more complexity to solve, both initially and in ongoing upkeep.

Re: Citi Missed a Fat Finger

#22

Post by Aloha »

zerocrates wrote: post #18
Symmetry wrote: post #3

I'm sure there have been more expensive UX failures (Three Mile Island?) but if there's ever a Wikipedia list of them I'm sure this will be on it.

Therac-25 comes to mind...

Therac-25 was at least more deadly.

Also much more complex than a UX failure.

Re: Citi Missed a Fat Finger

#23

Post by ploxiln »

dkarl wrote: post #17

> However, as data from that external feed was unavailable, the price of the value of the Index instead defaulted to -1 To me, this reeks of a couple of bad experiences I had repeatedly as a back end developer coordinating with the front end: 1. Demanding that my services always returned correct-looking data, even if it wasn’t available, because they couldn’t (or wouldn’t) handle errors. The story always went like th…

Maybe not as bad, but similar ... I've had a frontend dev say they needed to see designs from the designer before they could add any kind of error message or indicator. The status-quo is just silence, the page just stops doing anything. This is not unusual for various SPAs I've seen around the web. It is unfortunately a typical experience to click something ... no response? click again? wait how long? refresh? hello?

Have the designer design it, sure, but until that happens, you have to show something for auth expired (tab suspended for a long while), connection failed (flaky wifi), or 5xx from service - as much as I may try to make that never happen, it is possible (database failover for a few seconds?), and the design can't just be "there shall be no errors", you can't decide that.

Re: Citi Missed a Fat Finger

#24

Post by Aloha »

mgkimsal wrote: post #10
blackeyeblitzar wrote: post #9

Earlier quoted context omitted.

This is a phenomenon in so many different jobs, and is called “alarm fatigue”. From https://www.nytimes.com/2010/08/01/weekinreview/01wald.html : > On Tuesday, the National Transportation Safety Board said that a crash last year on the Washington subway system that killed nine people had happened partly because train dispatchers had been ignoring 9,000 alarms per week. Air traffic controllers, nuclear plant operators…

and... 'fixing' it can cause its own issues. I've been involved in some systems that routinely spit out dozens of warnings per day. After 'fixing' some of them - maybe getting a system down to a few per week, bug reports started coming in that the system was 'broken'. Because people noticed the boxes and messages they routinely ignored were gone, and this must be a problem. Lots of re-education may need to happen on…

I work on a system with notoriously noisy alarming and messaging logging - my first step is to tell the customer to ignore all of it.

I tell them to do two things to tell them how to check system health.

1. Is it working normally? if so, then nothing is wrong.

2. Walk by the hardware once a week and check for idiot lights, check the system dashboard if one of the idiot lights is on. If no error is found, its a hardware problem (failed supply, drive, fan), if an error is found it's a failed server - either way, call us.

There are a handful of cases not fixed by this, but they're limited to time sync failures - which are critical, but generates no user facing alarms.

Re: Citi Missed a Fat Finger

#25

Post by maicro »

tristor wrote: post #19
mgkimsal wrote: post #10

Earlier quoted context omitted.

and... 'fixing' it can cause its own issues. I've been involved in some systems that routinely spit out dozens of warnings per day. After 'fixing' some of them - maybe getting a system down to a few per week, bug reports started coming in that the system was 'broken'. Because people noticed the boxes and messages they routinely ignored were gone, and this must be a problem. Lots of re-education may need to happen on…

This triggered a repressed memory. I was once on an ops team responsible for an application that had been turned over to us written in Java. The application ran in Tomcat, and prior to turning it over to us, the external vendor had "helpfully" built some monitoring and alerting for the application. One of the metrics they monitored was how many errors per minute and per hour were happening the application's logs. Whe…

"The errors let you know it's working; it's when the errors stop you have to worry."

Re: Citi Missed a Fat Finger

#27

Post by coldpie »

z_zetetic_z wrote: post #20

I've been in the room when it happened. In one memorable incident, a junior trader I sat opposite bought 100 bn Japanese Yen instead of 10 bn. His excuse? "I lost track of the zeros". We'd only hired the guy because he had a PhD and "PhD's are smart" - lol. In his defense the UI was awful. He left soon after...

Japanese uses different "groups" of powers of ten, maybe that tripped him up?

1; 10; 100; 1 x 1000; 10 x 1000

versus

1; 10; 100; 1000; 10000; 1 x 10000; 10 x 10000; 100 x 10000

https://en.wikipedia.org/wiki/Japanese_numerals#Large_number...

Re: Citi Missed a Fat Finger

#28

Post by bombcar »

ryandrake wrote: post #15
blackeyeblitzar wrote: post #9

Earlier quoted context omitted.

This is a phenomenon in so many different jobs, and is called “alarm fatigue”. From https://www.nytimes.com/2010/08/01/weekinreview/01wald.html : > On Tuesday, the National Transportation Safety Board said that a crash last year on the Washington subway system that killed nine people had happened partly because train dispatchers had been ignoring 9,000 alarms per week. Air traffic controllers, nuclear plant operators…

I always wondered: Since Alarm Fatigue is well known, why do UX designers and/or product owners keep insisting on adding new alarms for things? Surely we know they increasingly don't work, the more you have.

Because if someone dies because you didn't alarm an alarm, you get yelled at.

If someone dies because they ignored an alarm you did alarm, that's on them.

Re: Citi Missed a Fat Finger

#29

Post by z_zetetic_z »

coldpie wrote: post #27
z_zetetic_z wrote: post #20

I've been in the room when it happened. In one memorable incident, a junior trader I sat opposite bought 100 bn Japanese Yen instead of 10 bn. His excuse? "I lost track of the zeros". We'd only hired the guy because he had a PhD and "PhD's are smart" - lol. In his defense the UI was awful. He left soon after...

Japanese uses different "groups" of powers of ten, maybe that tripped him up? 1; 10; 100; 1 x 1000; 10 x 1000 versus 1; 10; 100; 1000; 10000; 1 x 10000; 10 x 10000; 100 x 10000 https://en.wikipedia.org/wiki/Japanese_numerals#Large_number...

No, the gui input field das just 100000000000 vs 10000000000. Oops

Re: Citi Missed a Fat Finger

#30

Post by jh00ker »

sickofparadox wrote: post #4

"You get 711 alerts, you only see 18 of them, you are like “ehh 18 alerts is pretty much the normal number,” you override them all without reading." I believe this is called an 'alert that cried wolf'.

Humans are bad at processing large sets of similar data occurring repeatedly.

It comes up everywhere:

- On-Call alerts received by my engineering team for our microservices that usually self-resolve result in the first action being taken by the engineers to be "just wait and see if it's a false alarm." We work to reduce the number alerts overall to reduce the noise.

- I'm reminded of "Cigna saves millions by having its doctors reject claims without reading them" https://news.ycombinator.com/item?id=35304017 In order to keep up with the claims, they just auto-rejected them to see if they were appealed in a way to filter out the "noise."

- We hear so much about Tesla's "FSD (Supervised)" and it's request that drivers "don't become complacent" but it happens anyway. After enough time behind the wheel with FSD enabled, we are swayed by the string of successes to become overly trusting of the tech.

Post reply on HN