Cathy O’Neil on Weapons of Math Destruction • Hacker News Board

Re: Cathy O’Neil on Weapons of Math Destruction

#11

Post by anthony_james » Wed, Oct 05, 2016, 2:46 PM UTC

mcherm wrote: post #9

pliny wrote: post #7

Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information.

This isn't a black-and-white choice between including peripheral information into credit lending. When banks and lenders make a deliberate decision to ignore information that could allow them to be more accurate with lending, those costs are passed down to users - and although it may not hurt the typical HN user, marginalized credit-seekers can literally have their hopes of home-ownership or education denied because of a bank's choice to be ignorant but considerate. Neither choice is completely without consequences.

Re: Cathy O’Neil on Weapons of Math Destruction

#12

Post by leonidr » Wed, Oct 05, 2016, 3:10 PM UTC

One wonders about her opinion on the cost and benefits simulations that lead to the Affordable Care Act.

Re: Cathy O’Neil on Weapons of Math Destruction

#13

Post by robocaptain » Wed, Oct 05, 2016, 3:24 PM UTC

For anyone who is a fan of Cathy and wants to hear more, she is a co-host on the Slate Money podcast[0] (along with the amazing Felix Salmon) which covers a lot of these topics.

Although not ONLY these types of topics, I should say. It's a great nerdy finance podcast.

[0] http://www.slate.com/articles/podcasts/slate_money.html

Re: Cathy O’Neil on Weapons of Math Destruction

#14

Post by wfo » Wed, Oct 05, 2016, 3:45 PM UTC

anthony_james wrote: post #11

mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information. This isn't a black-and-white choice between including peripheral information into credit lending. When banks and lenders make a deliberate decision to…

It's true -- we also, however, ban firing based on race or sex. Purportedly, in the view of the managers who would wish to do so, their company would be more efficient if they were allowed to hire and fire whomever they wish (i.e. only employ white people, for example) -- perhaps they believe, in their racist world view, that having a company full of one race would create workplace unity, etc.

We could just let the market decide, and since we know the antiracist hypothesis is true, racist companies would miss on a huge amount of qualified labor and get beaten in a free market, forgetting for a moment that free markets are a myth and don't exist, and that the markets we do have aren't even close to efficient.

But we don't. We force companies to act a certain way even though they don't wish to because we are forcing ethical standards on them. This is, akin to your point at the expense of all of the workers not protected by the law. Every time we prevent a black person from being fired because he is black, it's at the expense of the white person who would take his job. The costs are passed down to white people. Neither choice is completely without consequences, yet we've made the one against racism.

Re: Cathy O’Neil on Weapons of Math Destruction

#15

Post by mcherm » Wed, Oct 05, 2016, 4:08 PM UTC

anthony_james wrote: post #11

mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information. This isn't a black-and-white choice between including peripheral information into credit lending. When banks and lenders make a deliberate decision to…

> It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information.

That's true. If we didn't give credit to more than a few outstanding black folks, then some poor whites could get credit cards at lower rates.

The most important thing to realize here is that DISCRIMINATION CAN WORK. If everyone agrees that redheads are no good, and everyone is extra careful about lending money to redheads and extra reluctant to hire redheads and extra-strict when deciding how to prosecute and sentence redheads, then network effects will make it a self-fulfilling prophecy. A redhead will be more likely to get caught up in criminal proceedings, will be more likely to get fired (or not hired in the first place), and therefore will be more likely to default on their loans.

For the most part, society has decided that this is either a moral outrage or a case of tragedy of the commons. From the moral point of view we say it's just not ethical to discriminate against people based on race, sex, family, and such. From a purely utilitarian point of view we can say that discrimination can benefit one party at a cost born by all of society. As is usual with tragedy of the commons situations, we can repair the problem with regulation. Regardless of whether you prefer the moral approach or the utilitarian one, there is a pretty strong case to be made that it is GOOD (on a society-wide basis) to give better deals to some (those marginal whites) than others (those marginal blacks) by prohibiting the use of certain information in granting credit.

Re: Cathy O’Neil on Weapons of Math Destruction

#16

Post by pron » Wed, Oct 05, 2016, 4:08 PM UTC

pliny wrote: post #7

Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…

> Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits

Before you choose to make a tradeoff, you should know what it is that you're trading off. If anything, much stronger forces than this book are pulling in the other direction.

Re: Cathy O’Neil on Weapons of Math Destruction

#17

Post by anthony_james » Wed, Oct 05, 2016, 4:09 PM UTC

wfo wrote: post #14

anthony_james wrote: post #11

Earlier quoted context omitted.

It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information. This isn't a black-and-white choice between including peripheral information into credit lending. When banks and lenders make a deliberate decision to…

It's true -- we also, however, ban firing based on race or sex. Purportedly, in the view of the managers who would wish to do so, their company would be more efficient if they were allowed to hire and fire whomever they wish (i.e. only employ white people, for example) -- perhaps they believe, in their racist world view, that having a company full of one race would create workplace unity, etc. We could just let the m…

I think this case is slightly more complicated because the demographics of marginalized credit-seekers tend to be minorities. So by catering to minority groups by keeping their ethnic or racial information private, we may hurt minority groups financially. In some of the situations you mention, we force ethical standards on businesses, but ideally they don't hurt other minority groups in the process.

In this situation, the question is between whether ensuring privacy is more ethical than ensuring access to capital - and this is almost entirely focused at minority groups. If we assume that banks can make more efficient and competitive lending transactions given more demographic information, then denying that information raises the ceiling on financial capital for those marginalized groups.

As of now, I don't have a definitive answer. Although I think it would be beneficial to examine how certain data impacts credit-lending and move from there. A lot of these concerns may be moot if the information in question isn't even relevant to credit lending.

Re: Cathy O’Neil on Weapons of Math Destruction

#18

Post by mhaymo » Wed, Oct 05, 2016, 4:16 PM UTC

Great podcast. It's largely about the fallacy that seemingly intractable problems can be solved with simple (and proprietary) machine learning models, and the damage we do by buying into it. In particular I don't know how the "recidivism risk scores" she rails against can be defended - how can a machine prediction based on a quiz be a better predictor than the judgment of a qualified judge who has presided over the defendant's trial?

Re: Cathy O’Neil on Weapons of Math Destruction

#19

Post by pvg » Wed, Oct 05, 2016, 4:17 PM UTC

mcherm wrote: post #9

pliny wrote: post #7

Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria.

You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. They don't because laws were passed prohibiting them from doing so.

Re: Cathy O’Neil on Weapons of Math Destruction

#20

Post by h4nkoslo » Wed, Oct 05, 2016, 4:32 PM UTC

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dissonance it's important to remain appropriately ignorant and not investigate anything where you might discover something "problematic".

You can tell she's interested in preventing knowledge from how she handled her job to predict effectiveness of homelessness services - she actively decided to not use particular variables on the grounds they might show a result she was uncomfortable with. That isn't an issue of "being aware of the limitations of machine learning", it's intentional ignorance.

http://www.slate.com/articles/technology/future_tense/2016/0...