Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attacking discrimination with smarter machine learning

research.google.com

Post reply on HN
Print view View original

31–40 of 201 posts

Re: Attacking discrimination with smarter machine learning

#31

Post by adrianN »

tptacek wrote: post #24
mvindahl wrote: post #22

Well. At the end of the day, the companies will pick thresholds and rates to maximize their profits, based upon the data that they have available. They don't know every detail of our personal lives -- which would also be kind of unsettling -- so they have to resort to a simplified picture. Simplifications are always prejudiced at the individual level but if the prejudice is reflected in the numbers, it's balanced. It…

The difference with machine learning is that the model isn't designed by humans, through an actuarial process we can keep our brains wrapped around. It's a black box. We are OK attributing "crash risk" to young male drivers, because we can observe both that they are as a cohort statistically likely to crash and also understand why that would be the case. On the other hand, we're not comfortable with the idea that a c…

You're assuming that human brains are good at differentiating between irrelevant correlation and causation. I don't think that's right.

Re: Attacking discrimination with smarter machine learning

#32

Post by tjic »

tptacek wrote: post #17
beejiu wrote: post #12

Presumably it means unlawful or unethical discrimination, since classification without discrimination doesn't make sense.

It's referring to the much-talked-about effect of emergent discrimination, where the model fitting process has the effect of amplifying the status quo, despite the fact that the status quo is informed in large part by structural injustices. For (oversimplified) instance: poor black people represent a cohort of loan applicants likely to default, and the model fitting process may go a step worse and attribute "default…

> despite the fact that the status quo is informed in large part by structural injustices.

This is presented as if it's an unambiguous fact, when it's largely a political stance.

Re: Attacking discrimination with smarter machine learning

#34

Post by tptacek »

tjic wrote: post #29
tptacek wrote: post #24

Earlier quoted context omitted.

The difference with machine learning is that the model isn't designed by humans, through an actuarial process we can keep our brains wrapped around. It's a black box. We are OK attributing "crash risk" to young male drivers, because we can observe both that they are as a cohort statistically likely to crash and also understand why that would be the case. On the other hand, we're not comfortable with the idea that a c…

Please don't say "we". Not everyone shares your politics. I'm perfectly happy with algorithms detecting that certain people are more likely to be safe drivers than average, and giving them lower rates, and concentrating premiums on the groups more likely to be in accidents, even if I don't understand why Armenians (in your example) get in more crashes.

You've just defined away the problem. It's not that we're not OK with computers accounting for the idea that Armenians get into more car accidents (we might or might not be). It's that we don't know if that's actually the case, because ML-generated models aren't that simple (if they were, we wouldn't need ML, just a bunch of actuaries using R).

The distinction is that with young male drivers, we have two supporting classes of information:

* A clear statistical observation

* A conceptual understanding of why the observation is likely to be valid

With machine learning, we might have neither of these classes of information.

Re: Attacking discrimination with smarter machine learning

#35

Post by reader5000 »

tptacek wrote: post #24
mvindahl wrote: post #22

Well. At the end of the day, the companies will pick thresholds and rates to maximize their profits, based upon the data that they have available. They don't know every detail of our personal lives -- which would also be kind of unsettling -- so they have to resort to a simplified picture. Simplifications are always prejudiced at the individual level but if the prejudice is reflected in the numbers, it's balanced. It…

The difference with machine learning is that the model isn't designed by humans, through an actuarial process we can keep our brains wrapped around. It's a black box. We are OK attributing "crash risk" to young male drivers, because we can observe both that they are as a cohort statistically likely to crash and also understand why that would be the case. On the other hand, we're not comfortable with the idea that a c…

I dont think most ML algos are as black box as you think, and if there were such an algo that regularly had a problem with finding "irrelevant correlations" then it wouldnt be a very good ML algo and people wouldn't use it.

Re: Attacking discrimination with smarter machine learning

#36

Post by adrianN »

d--b wrote: post #8

Nice to see that the debate has reached the ears of the main people working on this field. What is important to note here is that we need to tweak the mathematical model to the culture we want to achieve. In other words, the objective function of the optimization problem needs not only match the current state of the world, and provide an hindsight in one's own economic interests, it also needs to take into account th…

Tweaking the models to lead towards the world we'd like to live in seems like a noble goal, but I wonder how you should incentivize this. If discriminatory models provide higher profits, the idealistic models won't be used.

Re: Attacking discrimination with smarter machine learning

#37

Post by pdkl95 »

tptacek wrote: post #24
mvindahl wrote: post #22

Well. At the end of the day, the companies will pick thresholds and rates to maximize their profits, based upon the data that they have available. They don't know every detail of our personal lives -- which would also be kind of unsettling -- so they have to resort to a simplified picture. Simplifications are always prejudiced at the individual level but if the prejudice is reflected in the numbers, it's balanced. It…

The difference with machine learning is that the model isn't designed by humans, through an actuarial process we can keep our brains wrapped around. It's a black box. We are OK attributing "crash risk" to young male drivers, because we can observe both that they are as a cohort statistically likely to crash and also understand why that would be the case. On the other hand, we're not comfortable with the idea that a c…

> The difference with machine learning is that the model isn't designed by humans

Even if that was true[1], where are you getting the data from? Choosing which types of data to use is just as important as the model.

[1] as others have already pointed out, it depends on the technique/etc

Re: Attacking discrimination with smarter machine learning

#38

Post by beejiu »

mvindahl wrote: post #22

Well. At the end of the day, the companies will pick thresholds and rates to maximize their profits, based upon the data that they have available. They don't know every detail of our personal lives -- which would also be kind of unsettling -- so they have to resort to a simplified picture. Simplifications are always prejudiced at the individual level but if the prejudice is reflected in the numbers, it's balanced. It…

Discrimination is far more nuanced than it first appears.

We accept that younger drivers pay more for car insurance. And that disabled people pay more for their health insurance. And that the government (in the UK) has a special business grants scheme for ethnic minority entrepreneurs.

Obviously discrimination depends on context. If a car insurance firm had a special policy for ethnic minorities, people would (rightly) be outraged. But in the context of a government intervention, based on the evidenced disadvantages that ethnic minorities face, discrimination is accepted.

Re: Attacking discrimination with smarter machine learning

#39

Post by BickNowstrom »

riskable wrote: post #11
amelius wrote: post #4

But how can we find out if a company uses ML in a non-discriminating way? If we cannot see it, and check it, then there is no incentive for companies to use it. My guess is that at most companies spending time on making an algorithm non-discriminating will be viewed as a waste of time and money.

I work for a big bank... As long as people freak out when banks are found to be discriminating they will do their best to not discriminate. Banks are built on trustworthiness. Having your bank's name in the headlines for discriminatory practices can have a severe negative impact on trustworthiness. They have a whole teams of people devoted to this topic and every year at most banks every employee has to learn about,…

Like the line in Fight Club [1] about recalls it is a simple formula:

Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk model, is a bank economically incentivized to not discriminate.

Then again, even if you take care to not directly discriminate, you will probably indirectly discriminate. For instance, in your credit risk model, remove the `gender` column from the features, and use the remaining features to try to predict it. If performance is better than random guessing, you are using proxy features (like `income`) to discriminate on `gender`. You will find that nearly every feature you use is correlated with `race`. Now what? Throw away all these features and let your competition eat your lunch?

Along the lines of what Pedro Domingos said [2], you can not solve the problem of discrimination by making poorer performing machine learning models that adhere to your view of what is ideal. Discrimination won't disappear because you made a model that makes you feel good. Want no discrimination of women? Work on closing the wage gap. Don't cripple your statistically correct ML models or sweep the discrimination under the rug, covered by correlated variables.

It is not so much carelessness, as it is the nature of the beast. And banks remain in business by how much they can trust their customers first and foremost, trustworthiness by customers is a second (and customer trust is very much malleable: It is the perception of trust, not objective trust like "can we trust this customer to pay back their loan").

It also depends on how you (mathematically) define "fairness" [3]. You can define fairness in ways that still allow you to discriminate.

[1] A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.

[2] https://www.youtube.com/watch?v=furfdqtdAvc

[3] https://algorithmicfairness.wordpress.com/2016/09/26/on-the-... https://arxiv.org/abs/1609.07236

Re: Attacking discrimination with smarter machine learning

#40

Post by riskable »

tptacek wrote: post #20
riskable wrote: post #11

Earlier quoted context omitted.

I work for a big bank... As long as people freak out when banks are found to be discriminating they will do their best to not discriminate. Banks are built on trustworthiness. Having your bank's name in the headlines for discriminatory practices can have a severe negative impact on trustworthiness. They have a whole teams of people devoted to this topic and every year at most banks every employee has to learn about,…

That is a story that banks like to believe about themselves, but it's worth considering that emergent discrimination (and the broader bucket of emergent malfeasance) is a property of all complex systems, not just machine learning. So, for instance, Wells Fargo needed to do more than just hope that its trustworthiness would survive its incentive systems.

Not taking steps to prevent emergent discrimination or malfeasance is careless. Do you think the board of directors wants to find out that the CEO intentionally ignored calls to the ethics hotline? That he didn't take steps to ensure whistleblowers weren't punished?

The consequences of these things are very bad for stock prices! This is doubly true for banks which take reputational risk far more seriously than most businesses.

Post reply on HN