We investigated Amsterdam's attempt to build a 'fair' fraud detection model • Hacker News Board

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#81

Post by tripletao » Mon, Jun 16, 2025, 12:17 AM UTC

like_any_other wrote: post #72

tripletao wrote: post #30

Earlier quoted context omitted.

They correctly note the existence of a tradeoff, but I don't find their statement of it very clear. Ideally, a model would be fair in the senses that: 1. In aggregate over any nationality, people face the same probability of a false positive. 2. Two people who are identical except for their nationality face the same probability of a false positive. In general, it's impossible to achieve both properties. If the output…

> Ideally, a model would be fair in the senses that: 1. In aggregate over any nationality, people face the same probability of a false positive. Why? We've been told time and time again that 'nations' don't really exist, they're just recent meaningless social constructs [1]. And 'races' exist even less [2]. So why is it any worse if a model is biased on nation or race, than on left-handedness or musical taste or what…

I'm making a mathematical statement, not a moral one. I chose "nationality" as my input because the linked article focused on that, but the statement applies equally to any other input.

As already noted, any classifier better than a coin flip will disfavor some groups. The choice of which groups are acceptable to disfavor is political and somewhat arbitrary here. For example, these authors accept disfavoring people based on poverty ("sum of assets") or romantic relationship status ("single or partnered?"), but don't accept parenthood or nationality.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#82

Post by LorenPechtel » Mon, Jun 16, 2025, 2:29 AM UTC

Fraterkes wrote: post #60

LorenPechtel wrote: post #49

Why is there so much focus on "fair" even when reality isn't? Not all misdeeds are equally likely to be detected. What matter is minimizing the false positives and false negatives. But it sounds like they don't even have a base truth to be comparing it against, making the whole thing an exercise in bureaucracy.

Who says reality isnt fair? Isnt that up to us, the people inhabiting reality?

Reality gives some people an easier time than others.

If reality were fair there would be no need of a welfare system in the first place.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#83

Post by dannykwells » Mon, Jun 16, 2025, 4:18 AM UTC

Can someone explain to me - assuming they have enough data - why not train different models explicitly for each group / subgroup you want to model? You even could then just take the top N% of each group by score, effectively guaranteeing equal treatment for each group. Why would this not work?

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#84

Post by alternatex » Tue, Jun 17, 2025, 5:01 PM UTC

triceratops wrote: post #77

alternatex wrote: post #70

Earlier quoted context omitted.

I've never had any implication of my gender other than my name in any CV over the past decade. Who are these people who make a career history doc include gender-implicating data? And if there are such CVs, they should be stripped of such data before processing. The fraternity example is such a specific 1 in a 1000 case.

> I've never had any implication of my gender other than my name in any CV So you're not implying gender other than by implying gender? If humans can use names to classify people into genders, a model can do the same thing.

It's information that's easy to strip before running it through machine learning.

The implication in the parent comment is that CVs are inherently bound to gender and I cannot see that to be the case for most.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#85

Post by alternatex » Tue, Jun 17, 2025, 5:03 PM UTC

crote wrote: post #79

alternatex wrote: post #70

Earlier quoted context omitted.

I've never had any implication of my gender other than my name in any CV over the past decade. Who are these people who make a career history doc include gender-implicating data? And if there are such CVs, they should be stripped of such data before processing. The fraternity example is such a specific 1 in a 1000 case.

Just because you aren't aware of it, doesn't mean it isn't there. There are plenty of less on-the-nose examples a model can accidentally train itself on. Into horseriding? Probably a woman. Into motorcycles? Probably a man. Into musical theater? Probably a woman. Into football? Probably a man. Worked part-time for a few years in your 30s? Probably a woman. I could go on and on for hours, as there are relatively few h…

I feel like what will be left is only the things that are good indicators of professional competence. Of all things that are fluff in a CV, a hobbies section would probably be #1. Not to mention probably a red flag for any CV reviewer.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#86

Post by tanewishly » Wed, Jun 18, 2025, 10:42 PM UTC

tripletao wrote: post #80

tanewishly wrote: post #75

Earlier quoted context omitted.

> 2. Two people who are identical except for their nationality face the same probability of a false positive. That seems to fall afoul of the Base Rate Fallacy. Eg, consider 2 groups of 10,000 people and testing on A vs B. First group has 9,999 A and 1 B, second has 1 A and 9,999 B. Unless you make your test blatantly ineffective, you're going to have different false positive rates -- irrespectiveof the test's perfor…

The linked article already notes that model accuracy degraded after their reweighting, ultimately contributing to their abandonment of the project. (For completeness, they could also have considered nationality in the opposite direction, improving accuracy vs. nominally blind baseline at the cost of yet more disparate false positives; but that's so politically unacceptable that it's not even mentioned.) My point is t…

My point was that there is no test (or classifier) that can always guarantee that one definition of fairness by itself, irrespective of the base rate. If the classifier acts the same independent of base rate, there are always base rates (ie occurrence rates in the rates population) for which the classifier will fail the given definition.

That illustrates that the given definition cannot hold universally, irrespective of what classifier you dream up. Unless your classifier is not independent from the base rate - that is, a classifier that gets more lenient if there's more fraud in the group. That seems undesirable when considering fairness as a goal.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#87

Post by tripletao » Thu, Jun 19, 2025, 7:53 PM UTC

tanewishly wrote: post #86

tripletao wrote: post #80

Earlier quoted context omitted.

The linked article already notes that model accuracy degraded after their reweighting, ultimately contributing to their abandonment of the project. (For completeness, they could also have considered nationality in the opposite direction, improving accuracy vs. nominally blind baseline at the cost of yet more disparate false positives; but that's so politically unacceptable that it's not even mentioned.) My point is t…

My point was that there is no test (or classifier) that can always guarantee that one definition of fairness by itself, irrespective of the base rate. If the classifier acts the same independent of base rate, there are always base rates (ie occurrence rates in the rates population) for which the classifier will fail the given definition. That illustrates that the given definition cannot hold universally, irrespective…

> Unless your classifier is not independent from the base rate - that is, a classifier that gets more lenient if there's more fraud in the group.

If I understand correctly, that was the purpose of their "reweighting". To remove "bias" from their new model, they looked at the aggregate false positive rates by nationality, then applied explicit penalty to people with Western nationality to bring that FPR equal to non-Western.

I'd certainly agree that's undesirable, and my point was also to highlight that (since the article was reasonably clear on the type of fairness they achieved, but much less clear on the fairness they sacrificed).