Earlier quoted context omitted.
I worked on a project where one of the services was a model that decided whether to pay a medical bill. Before you start justified screams of horror, let me explain the simple honesty trick that ensured proper ethics, though I guess at cost of profit unacceptable to some corporations: The model could only decide between auto approving a repayment, or refer the bill to existing human staff. The entire idea was that th…
Mmmmhm, which means the humans now understand that they should be callous and cold. If they're not rubber stamping rejections all the time then the AI isn't doing anything useful by making a feed of easy-to-reject applications. The system will become evil even if it has humans in it because they have been given no power to resist the incentives
It helps, as you can suspect from "union" comment, that it wasn't an american health care insurance company.