Cathy O’Neil on Weapons of Math Destruction • Hacker News Board

Re: Cathy O’Neil on Weapons of Math Destruction

#21

Post by inimino » Wed, Oct 05, 2016, 4:55 PM UTC

pvg wrote: post #19

mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria. You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. They don't because laws were passed prohibiting them…

Those laws were passed because of cultural values that are widely shared. You make it sound as if bankers would prefer racist policies if only the law would allow it. GP's point is that even though most bankers do have such high-minded motivations, regulation is necessary to avoid bad actors from gaining a competitive advantage in a race to the bottom.

Re: Cathy O’Neil on Weapons of Math Destruction

#22

Post by throwaway40483 » Wed, Oct 05, 2016, 4:57 PM UTC

mhaymo wrote: post #18

Great podcast. It's largely about the fallacy that seemingly intractable problems can be solved with simple (and proprietary) machine learning models, and the damage we do by buying into it. In particular I don't know how the "recidivism risk scores" she rails against can be defended - how can a machine prediction based on a quiz be a better predictor than the judgment of a qualified judge who has presided over the d…

I think she addressed this by saying that the judges usually end up being more racist than the baked in racism in the algorithms.

Re: Cathy O’Neil on Weapons of Math Destruction

#23

Post by throwaway40483 » Wed, Oct 05, 2016, 5:00 PM UTC

robocaptain wrote: post #13

For anyone who is a fan of Cathy and wants to hear more, she is a co-host on the Slate Money podcast[0] (along with the amazing Felix Salmon) which covers a lot of these topics. Although not ONLY these types of topics, I should say. It's a great nerdy finance podcast. [0] http://www.slate.com/articles/podcasts/slate_money.html

Definitely one of the better (high wonkish/entertainment ratio) podcasts.

Re: Cathy O’Neil on Weapons of Math Destruction

#24

Post by inimino » Wed, Oct 05, 2016, 5:09 PM UTC

h4nkoslo wrote: post #20

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dissonance it's important to remain appropriately ignorant and not investigate anything where you might discover something "problematic". You can tell she's interested in preventing knowledge from how she handled her job to predict effectiveness of homelessness se…

Yes, "intentional ignorance" has value. Justice is blind. That's why some companies blind gender when screening applications, and why most interviewers aren't likely to ask a candidate about their religion or politics or sexual preferences.

Sometimes correlations are self-perpetuating, and it's better to not know about them when making decisions.

Re: Cathy O’Neil on Weapons of Math Destruction

#25

Post by powertower » Wed, Oct 05, 2016, 5:24 PM UTC

mcherm wrote: post #9

pliny wrote: post #7

Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

For every citizen to have an equal chance of success (assuming success is merit based), every citizen would have to have equal abilities.

And this is simply not true. Especially by race where IQ can vary 3 standard deviations or more.

Re: Cathy O’Neil on Weapons of Math Destruction

#26

Post by h4nkoslo » Wed, Oct 05, 2016, 5:30 PM UTC

inimino wrote: post #24

h4nkoslo wrote: post #20

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dissonance it's important to remain appropriately ignorant and not investigate anything where you might discover something "problematic". You can tell she's interested in preventing knowledge from how she handled her job to predict effectiveness of homelessness se…

Yes, "intentional ignorance" has value. Justice is blind. That's why some companies blind gender when screening applications, and why most interviewers aren't likely to ask a candidate about their religion or politics or sexual preferences. Sometimes correlations are self-perpetuating, and it's better to not know about them when making decisions.

Well, it only "has value" if you're worried about discovering the wrong thing. If you think you're going to discover the "correct" thing, then you're apparently morally free to use whatever variables you want.

Re: Cathy O’Neil on Weapons of Math Destruction

#27

Post by powertower » Wed, Oct 05, 2016, 5:32 PM UTC

inimino wrote: post #21

pvg wrote: post #19

Earlier quoted context omitted.

But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria. You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. They don't because laws were passed prohibiting them…

Those laws were passed because of cultural values that are widely shared. You make it sound as if bankers would prefer racist policies if only the law would allow it. GP's point is that even though most bankers do have such high-minded motivations, regulation is necessary to avoid bad actors from gaining a competitive advantage in a race to the bottom.

How is lending based on all qualifying factors a race to the bottom?

Re: Cathy O’Neil on Weapons of Math Destruction

#28

Post by panglott » Wed, Oct 05, 2016, 5:43 PM UTC

Some really interesting points in her negative review of Nate Silver's book:

"In baseball, a team can’t create bad or misleading data to game the models of other teams in order to get an edge. But in the financial markets, parties to a model can and do. ...Silver gives four examples what he considers to be failed models at the end of his first chapter, all related to economics and finance. But each example is actually a success (for the insiders) if you look at a slightly larger picture and understand the incentives inside the system. ...Silver confuses cause and effect. We didn’t have a financial crisis because of a bad model or a few bad models. We had bad models because of a corrupt and criminally fraudulent financial system.

...Silver has an unswerving assumption, which he repeats several times, that the only goal of a modeler is to produce an accurate model."

Her other examples are things like pharmaceuticals research. http://www.nakedcapitalism.com/2012/12/cathy-oneil-why-nate-...

Re: Cathy O’Neil on Weapons of Math Destruction

#29

Post by pvg » Wed, Oct 05, 2016, 5:55 PM UTC

inimino wrote: post #21

pvg wrote: post #19

Earlier quoted context omitted.

But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria. You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. They don't because laws were passed prohibiting them…

Those laws were passed because of cultural values that are widely shared. You make it sound as if bankers would prefer racist policies if only the law would allow it. GP's point is that even though most bankers do have such high-minded motivations, regulation is necessary to avoid bad actors from gaining a competitive advantage in a race to the bottom.

I'm not making it sound like that, that was precisely the case. Hence the laws.

Re: Cathy O’Neil on Weapons of Math Destruction

#30

Post by bobcostas55 » Wed, Oct 05, 2016, 6:03 PM UTC

mcherm wrote: post #9

pliny wrote: post #7

Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

Consider an algorithm that perfectly predicts defaults. If such an algorithm existed, non-defaulters would be able to borrow at the risk-free rate. Some people who would be unfairly denied loans in the current regime would be able to get them. This is clearly desirable.

Improving your ability to predict defaults is simply moving in the direction of that perfect algorithm. The inputs you use, whatever your emotional reaction to them, are irrelevant.

By not optimizing your predictions to the fullest extent possible, some people are worse off just as some others are better off. There is always such a trade-off. There is absolutely nothing special about the status quo. The idea that one side of this trade-off is somehow more ethical than the other is completely absurd.