Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history and future of credit scores

economist.com

Post reply on HN
Print view View original

11–20 of 72 posts

Re: A brief history and future of credit scores

#11

Post by nulbyte »

harryh wrote: post #7
dvdbloc wrote: post #5

Earlier quoted context omitted.

I find this interesting because when I applied for a mortgage at Wells Fargo they asked me for my race in the application process.

They probably have to report on loan approval rates by race to demonstrate a lack of bias in their underwriting.

Yes. For example, mortgage lenders in the US are required to obtain and report this.[1] If applying in person the applicant chooses not to furnish the information, the lender is required to guess based on observation.

[1]: http://mortgageporter.com/2010/12/your-ethnicity-and-race-on...

Re: A brief history and future of credit scores

#12

Post by mieseratte »

emadelwany wrote: post #9
dd36 wrote: post #6

Earlier quoted context omitted.

I get confused by those they claim some of these ML or NN algos are black boxes and so could be breaking the law. If you’re not inputting illegal info (like race, sex, national origin, religion, name, etc. or corollaries), then it’s not making its risk assessment on that basis. All you have to do is look at the inputs. It isn’t unknowable whether or not it’s breaking the law.

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

Re: A brief history and future of credit scores

#13

Post by Arcsech »

mieseratte wrote: post #12
emadelwany wrote: post #9

Earlier quoted context omitted.

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

Yes, for one example, a system used to recommend sentencing based on how likely a criminal was to re-offend was found to have been heavily biased based on race, despite not being given explicit information about race[1].

Edit: It’s unclear if this system is based on ML, but I think the point stands - if a system can be manually tuned to do this, the risk exists of this happening to a trained model as well. Given other cases of ML models learning an incorrect behavior (such as the system intended to detect skin cancer that ended up being a fancy ruler detector [2]).

[1]: https://www.propublica.org/article/machine-bias-risk-assessm... [2]: https://www.thedailybeast.com/why-doctors-arent-afraid-of-be...

Re: A brief history and future of credit scores

#14

Post by harryh »

Arcsech wrote: post #13
mieseratte wrote: post #12

Earlier quoted context omitted.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

Yes, for one example, a system used to recommend sentencing based on how likely a criminal was to re-offend was found to have been heavily biased based on race, despite not being given explicit information about race[1]. Edit: It’s unclear if this system is based on ML, but I think the point stands - if a system can be manually tuned to do this, the risk exists of this happening to a trained model as well. Given othe…

That is a fine answer to mieseratte's question but I feel compelled to mention that the propublica article you link to is...pretty bad.

Here's a pretty good summary of why:

https://www.chrisstucchio.com/blog/2016/propublica_is_lying....

Re: A brief history and future of credit scores

#15

Post by AmericanChopper »

mieseratte wrote: post #12
emadelwany wrote: post #9

Earlier quoted context omitted.

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

Using zip codes to implement discriminatory policies isn’t ML, but has almost certainly occurred. It’s referred to as redlining, and has been around for a while. The New Deal had some pretty indefensible redlining conditions. A lot of people will say the practice continues to this day, but the modern examples are a lot more open to interpretation than some of the historical ones are.

Re: A brief history and future of credit scores

#16

Post by jetrink »

mieseratte wrote: post #12
emadelwany wrote: post #9

Earlier quoted context omitted.

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon's recruiting process.

1. https://www.reuters.com/article/us-amazon-com-jobs-automatio...

Re: A brief history and future of credit scores

#17

Post by swsieber »

There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race).

Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?

Re: A brief history and future of credit scores

#18

Post by Spooky23 »

dd36 wrote: post #6
dvdbloc wrote: post #5

Earlier quoted context omitted.

I find this interesting because when I applied for a mortgage at Wells Fargo they asked me for my race in the application process.

I get confused by those they claim some of these ML or NN algos are black boxes and so could be breaking the law. If you’re not inputting illegal info (like race, sex, national origin, religion, name, etc. or corollaries), then it’s not making its risk assessment on that basis. All you have to do is look at the inputs. It isn’t unknowable whether or not it’s breaking the law.

Redlining was basically part of the law, as FHA historically required higher down payments for “less desirable”, aka non-white neighborhoods.

In the countryside or exurb, a similar dynamic arises... “the wrong side of the tracks”. Even without the racial dimension, the trailer parks and the horse farms tend to be in easily identified geographically distinct areas.

In hot markets in my area it’s easy to see. The “desirable” areas surge ahead of the baseline tax assessment comps, and the “undesirable” do not. You can map it in a stable area and it lines up with the old FHA maps.

Decades of a market like that leads to those conditions staying around for awhile. As multi-family real estate is getting appealing to people with money, I think we’ll finally see that legacy fade away.

Re: A brief history and future of credit scores

#19

Post by mieseratte »

jetrink wrote: post #16
mieseratte wrote: post #12

Earlier quoted context omitted.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon'…

I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.

Re: A brief history and future of credit scores

#20

Post by mieseratte »

AmericanChopper wrote: post #15
mieseratte wrote: post #12

Earlier quoted context omitted.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

Using zip codes to implement discriminatory policies isn’t ML, but has almost certainly occurred. It’s referred to as redlining, and has been around for a while. The New Deal had some pretty indefensible redlining conditions. A lot of people will say the practice continues to this day, but the modern examples are a lot more open to interpretation than some of the historical ones are.

Was the New Deal redlining out-and-out discrimination or a matter of accidentally creating conditions, i.e. based on bad intelligence, that resulted in discrimination. I always understood it to be the former.
Post reply on HN