Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We investigated Amsterdam's attempt to build a 'fair' fraud detection model

lighthousereports.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#71

Post by luckylion »

drdaeman wrote: post #62
luckylion wrote: post #56

Earlier quoted context omitted.

> It is considered to be unethical to discriminate upon things a person has no control over, such as their nationality, gender, age or natural hair color. Nationality and natural hair color I understand, but age and gender? A lot of behaviors are not evenly distributed. Riots after a football match? You're unlikely to find a lot of elderly women (and men, but especially women) involved. Someone is fattening a child?…

> A lot of behaviors are not evenly distributed. That’s true. But the idea is that feeding it to a system as an input could be considered unethical, as one cannot control their age. Even though there’s a valid correlation. > If you assume perfect free will, sure. But do you? I’m not. If this matters, I’m actually currently persuaded that free will doesn’t exist. Which doesn’t change that if one buys a car, its make i…

I get where you're coming from, but I very much doubt it's true RE car makes (and many similar things). There's a reason men and women have very distinct buying habits. E.g. men are ~4x more likely to buy a motorcycle. Individual decisions with that large a discrepancy between groups aren't individual decisions.

Can a young male really change their risk-tolerance or their innate drive to secure their place in the world (which will probably affect both their likelihood to buy sports cars and commit certain crimes)? I don't think we can pretend that everyone from toddler to granny is the same _and_ use any data to solve crimes / detect fraud.

In the end it comes down to where we draw the line between "person can't change this, so it's invalid to consider", "we don't believe it's linked, so it's invalid to consider" and "this is free will, so it's a valid signal", and I haven't seen a line that doesn't feel arbitrary ("I don't like that group, so their thing is free will, but I like this group, so their thing isn't") and is useful.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#72

Post by like_any_other »

tripletao wrote: post #30
tbrownaw wrote: post #23

> But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies. There's a huge problem with people trying to use umbrella usage to predict flooding. Some people are trying to develop a computer model that uses rainfall instead, but watchdog groups have raised concerns that rainfall may be used as a proxy for umbrella usage. (It seems rather strange…

They correctly note the existence of a tradeoff, but I don't find their statement of it very clear. Ideally, a model would be fair in the senses that: 1. In aggregate over any nationality, people face the same probability of a false positive. 2. Two people who are identical except for their nationality face the same probability of a false positive. In general, it's impossible to achieve both properties. If the output…

> Ideally, a model would be fair in the senses that: 1. In aggregate over any nationality, people face the same probability of a false positive.

Why? We've been told time and time again that 'nations' don't really exist, they're just recent meaningless social constructs [1]. And 'races' exist even less [2]. So why is it any worse if a model is biased on nation or race, than on left-handedness or musical taste or what brand of car one drives? They're all equally meaningless, aren't they?

[1] https://www.reddit.com/r/AskHistorians/comments/18ubjpv/the_...

[2] https://www.scientificamerican.com/article/race-is-a-social-...

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#73

Post by Viliam1234 »

GardenLetter27 wrote: post #67

> None of these features explicitly referred to an applicant’s gender or racial background, as well as other demographic characteristics protected by anti-discrimination law. But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies. What's the problem with this? It isn't racism, it's literally just Bayes' Law.

> It isn't racism, it's literally just Bayes' Law.

That may be logically correct, but the law is above logic. Sometimes applying Bayes' Law is legally considered racism.

https://en.wikipedia.org/wiki/Disparate_impact

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#75

Post by tanewishly »

tripletao wrote: post #30
tbrownaw wrote: post #23

> But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies. There's a huge problem with people trying to use umbrella usage to predict flooding. Some people are trying to develop a computer model that uses rainfall instead, but watchdog groups have raised concerns that rainfall may be used as a proxy for umbrella usage. (It seems rather strange…

They correctly note the existence of a tradeoff, but I don't find their statement of it very clear. Ideally, a model would be fair in the senses that: 1. In aggregate over any nationality, people face the same probability of a false positive. 2. Two people who are identical except for their nationality face the same probability of a false positive. In general, it's impossible to achieve both properties. If the output…

> 2. Two people who are identical except for their nationality face the same probability of a false positive.

That seems to fall afoul of the Base Rate Fallacy. Eg, consider 2 groups of 10,000 people and testing on A vs B. First group has 9,999 A and 1 B, second has 1 A and 9,999 B. Unless you make your test blatantly ineffective, you're going to have different false positive rates -- irrespectiveof the test's performance.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#76

Post by anonym29 »

Viliam1234 wrote: post #73
GardenLetter27 wrote: post #67

> None of these features explicitly referred to an applicant’s gender or racial background, as well as other demographic characteristics protected by anti-discrimination law. But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies. What's the problem with this? It isn't racism, it's literally just Bayes' Law.

> It isn't racism, it's literally just Bayes' Law. That may be logically correct, but the law is above logic. Sometimes applying Bayes' Law is legally considered racism. https://en.wikipedia.org/wiki/Disparate_impact

Legal frameworks can indeed contradict mathematical optimization functions, statistical patterns exist independent of our social preferences about them, and aggregate behavioral differences between groups (whatever their causes) will produce disparate algorithmic outcomes when accurately measured.

If certain demographic groups legitimately have higher base rates of welfare errors (due to language barriers, unfamiliarity with bureaucratic systems, economic desperation, or other factors), then an accurate algorithm will necessarily produce disparate outcomes.

If we dig deeper, there are three different underlying questions that are attempting to be addressed by the authors of this "fair" fraud detection system -

1. Do group differences in fraud rates actually exist?

2. What mechanisms drive these differences?

3. Should algorithms optimize for accuracy or equality of outcomes?

The article conflates these, treating disparate outcomes as presumptive evidence of algorithmic bias rather than potentially accurate detection of real differences.

Pattern recognition that produces disparate outcomes isn't necessarily inherently "broken", it may be simply be accurately detecting real underlying patterns whose causes are uncomfortable to acknowledge or difficult to address through algorithmic modifications alone.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#77

Post by triceratops »

alternatex wrote: post #70
crote wrote: post #69

Earlier quoted context omitted.

Let's say you are making a model to judge job applicants. You are aware that the training data is biased in favor of men, so you remove all explicit mentions of gender from their CVs and cover letters. Upon evaluation, your model seems to accept everyone who mentions a "fraternity" and reject anyone who mentions a "sorority". Swapping out the words turns a strong reject into a strong accept, and vice versa. But you r…

I've never had any implication of my gender other than my name in any CV over the past decade. Who are these people who make a career history doc include gender-implicating data? And if there are such CVs, they should be stripped of such data before processing. The fraternity example is such a specific 1 in a 1000 case.

> I've never had any implication of my gender other than my name in any CV

So you're not implying gender other than by implying gender? If humans can use names to classify people into genders, a model can do the same thing.

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#78

Post by thunderfork »

alternatex wrote: post #70
crote wrote: post #69

Earlier quoted context omitted.

Let's say you are making a model to judge job applicants. You are aware that the training data is biased in favor of men, so you remove all explicit mentions of gender from their CVs and cover letters. Upon evaluation, your model seems to accept everyone who mentions a "fraternity" and reject anyone who mentions a "sorority". Swapping out the words turns a strong reject into a strong accept, and vice versa. But you r…

I've never had any implication of my gender other than my name in any CV over the past decade. Who are these people who make a career history doc include gender-implicating data? And if there are such CVs, they should be stripped of such data before processing. The fraternity example is such a specific 1 in a 1000 case.

[dead]

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#79

Post by crote »

alternatex wrote: post #70
crote wrote: post #69

Earlier quoted context omitted.

Let's say you are making a model to judge job applicants. You are aware that the training data is biased in favor of men, so you remove all explicit mentions of gender from their CVs and cover letters. Upon evaluation, your model seems to accept everyone who mentions a "fraternity" and reject anyone who mentions a "sorority". Swapping out the words turns a strong reject into a strong accept, and vice versa. But you r…

I've never had any implication of my gender other than my name in any CV over the past decade. Who are these people who make a career history doc include gender-implicating data? And if there are such CVs, they should be stripped of such data before processing. The fraternity example is such a specific 1 in a 1000 case.

Just because you aren't aware of it, doesn't mean it isn't there. There are plenty of less on-the-nose examples a model can accidentally train itself on.

Into horseriding? Probably a woman. Into motorcycles? Probably a man. Into musical theater? Probably a woman. Into football? Probably a man. Worked part-time for a few years in your 30s? Probably a woman. I could go on and on for hours, as there are relatively few hobbies and interests which have a truly gender-neutral audience.

The problem isn't obvious bias. Everyone can see those and filtering them out is trivial. It's the subtle proxy values which are risky, as you have to be very careful to avoid accidentally training on those.

CVs should ideally indeed be stripped of such data, but how do you propose we verify that we stripped it of all potential proxies? And what's going to be left to train on after stripping?

Re: We investigated Amsterdam's attempt to build a 'fair' fraud detection model

#80

Post by tripletao »

tanewishly wrote: post #75
tripletao wrote: post #30

Earlier quoted context omitted.

They correctly note the existence of a tradeoff, but I don't find their statement of it very clear. Ideally, a model would be fair in the senses that: 1. In aggregate over any nationality, people face the same probability of a false positive. 2. Two people who are identical except for their nationality face the same probability of a false positive. In general, it's impossible to achieve both properties. If the output…

> 2. Two people who are identical except for their nationality face the same probability of a false positive. That seems to fall afoul of the Base Rate Fallacy. Eg, consider 2 groups of 10,000 people and testing on A vs B. First group has 9,999 A and 1 B, second has 1 A and 9,999 B. Unless you make your test blatantly ineffective, you're going to have different false positive rates -- irrespectiveof the test's perfor…

The linked article already notes that model accuracy degraded after their reweighting, ultimately contributing to their abandonment of the project. (For completeness, they could also have considered nationality in the opposite direction, improving accuracy vs. nominally blind baseline at the cost of yet more disparate false positives; but that's so politically unacceptable that it's not even mentioned.)

My point is that even if we're willing to trade accuracy for "fairness", it's not possible for any classifier to satisfy both those definitions of fairness. By returning to human judgment they've obfuscated that problem but not solved it.

Post reply on HN