Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cathy O’Neil on Weapons of Math Destruction

econtalk.org

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Cathy O’Neil on Weapons of Math Destruction

#31

Post by mcherm »

pvg wrote: post #19
mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria. You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. They don't because laws were passed prohibiting them…

> You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics.

I work for such a lender. Yes, that is precisely what I am saying. Although I might call it "basic ethics" not "high-minded civic duty".

Typically now "you" (someone raising this issue, because I don't want to put words in your mouth) point out that corporations are not ethical and simply act to seek maximal profit. Then I point out that corporations have a culture, which may have values other than just profit, and that corporations are made up of individuals who ALSO have motivations other than just profit.

The company I work for also has resisted the urge to open accounts that customers didn't ask for (as recently revealed of Wells Fargo) and many other ethical deviations. I am sure we have our own sins, and should work to improve them. I agree that laws constraining corporate behavior are ONE good tool for managing corporate behavior, but it is not true that absent such laws there would be no other constraints.

Re: Cathy O’Neil on Weapons of Math Destruction

#32

Post by munificent »

mcherm wrote: post #9
pliny wrote: post #7

Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

In capitalist terms, a perfect market will optimize for the world as it is. Many people think of businesses as simply animals that compete in an immutable environment and may the best one win. And then, especially in the US, somehow that observation takes on the weight of moral certitude and now we think it's somehow right for businesses to do that.

But the world is not immutable. And if we silently allow companies to optimize for the unjust world of today, we make it harder to build a more just world for tomorrow.

Denying loans to people of color may be price optimal today, but that's true only to the degree that the world of today sucks. We should fix that, not optimize for it.

Re: Cathy O’Neil on Weapons of Math Destruction

#33

Post by panglott »

h4nkoslo wrote: post #20

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dissonance it's important to remain appropriately ignorant and not investigate anything where you might discover something "problematic". You can tell she's interested in preventing knowledge from how she handled her job to predict effectiveness of homelessness se…

No. Her concern in that case was that a predictive model that measured the effects of racism on some people would then be used to increase the racial discrimination against them. That is absolutely an ethical question. Her task was not merely to find out facts about the world, but to build systems that would impact the welfare of people in the world—and people in a very precarious position, no less.

Race is not a valid biological category; it is a social construct and a salient social category. & as such it remains the site of many unethical uses of power. At times measuring them can reify those injustices; at times it is essential in combating them.

Re: Cathy O’Neil on Weapons of Math Destruction

#34

Post by h4nkoslo »

panglott wrote: post #33
h4nkoslo wrote: post #20

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dissonance it's important to remain appropriately ignorant and not investigate anything where you might discover something "problematic". You can tell she's interested in preventing knowledge from how she handled her job to predict effectiveness of homelessness se…

No. Her concern in that case was that a predictive model that measured the effects of racism on some people would then be used to increase the racial discrimination against them. That is absolutely an ethical question. Her task was not merely to find out facts about the world, but to build systems that would impact the welfare of people in the world—and people in a very precarious position, no less. Race is not a val…

You've essentially restated what I said above. She declined to investigate certain variables because she was worried they would come out with the "wrong" effects (instead she chose to use near-perfect correlates of those variables, like zip code).

You can always decline to use a variable after the fact; she decided she didn't want to know the effect in the first place. Suppose the effect had been the opposite of what she suspected?

Re: Cathy O’Neil on Weapons of Math Destruction

#35

Post by inimino »

powertower wrote: post #27
inimino wrote: post #21

Earlier quoted context omitted.

Those laws were passed because of cultural values that are widely shared. You make it sound as if bankers would prefer racist policies if only the law would allow it. GP's point is that even though most bankers do have such high-minded motivations, regulation is necessary to avoid bad actors from gaining a competitive advantage in a race to the bottom.

How is lending based on all qualifying factors a race to the bottom?

Because to discriminate on some of those factors would sacrifice certain cultural values, such as equality and a fair shot for everyone, which we happen to value more highly than any marginal increase in lending efficiency.

Re: Cathy O’Neil on Weapons of Math Destruction

#36

Post by icebraining »

munificent wrote: post #32
mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

In capitalist terms, a perfect market will optimize for the world as it is. Many people think of businesses as simply animals that compete in an immutable environment and may the best one win. And then, especially in the US, somehow that observation takes on the weight of moral certitude and now we think it's somehow right for businesses to do that. But the world is not immutable. And if we silently allow companies t…

Sure, but there's still a question on how to fix that. Should we prevent these companies from applying more targeted models, and hence impose a tax on the random subset of consumers who happen to use the same services? Or should we take tax money, which is taken from everyone in a democratically decided way, to help people of color get loans?

Or to put it more specifically, why should a middle-class white family have to pay higher rates on their loan to help some family of color get a loan, while the rich white family that buys the house outright doesn't?

Shouldn't we help people of color using a system that distributes the costs in a widespread and progressive way?

Re: Cathy O’Neil on Weapons of Math Destruction

#37

Post by jkaptur »

bobcostas55 wrote: post #30
mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

Consider an algorithm that perfectly predicts defaults. If such an algorithm existed, non-defaulters would be able to borrow at the risk-free rate. Some people who would be unfairly denied loans in the current regime would be able to get them. This is clearly desirable. Improving your ability to predict defaults is simply moving in the direction of that perfect algorithm. The inputs you use, whatever your emotional r…

Given the absence of such a perfect algorithm, allowing or favoring certain types of errors over others is absolutely a decision with ethical consequences.

Re: Cathy O’Neil on Weapons of Math Destruction

#38

Post by jxramos »

anthony_james wrote: post #11
mcherm wrote: post #9

Earlier quoted context omitted.

> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…

It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information. This isn't a black-and-white choice between including peripheral information into credit lending. When banks and lenders make a deliberate decision to…

good pithy way to put it, "ignorant but considerate". I think our society entering into the whole data space will unveil more and more ACTUAL disparities and inequalities with such accuracy that economists may very well introduce concepts and notions such as the hidden costs to willful ignorance. Perhaps call it data omission costs, blacklist data costs, model error from black list data omission. Stuff like that. You hit the nail on the head, neither choice is without consequences.

Re: Cathy O’Neil on Weapons of Math Destruction

#39

Post by pvg »

mcherm wrote: post #31
pvg wrote: post #19

Earlier quoted context omitted.

But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria. You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. They don't because laws were passed prohibiting them…

> You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics. I work for such a lender. Yes, that is precisely what I am saying. Although I might call it "basic ethics" not "high-minded civic duty". Typically now "you" (someone raising this issue, because I don't want to put words in your mouth) point out that corporations are not ethical and simply…

I'm sure you and your coworkers are people of the highest moral fiber but my point is you're congratulating yourself for not doing something plain illegal. And it's illegal now because people worked very hard to make it so and before that lenders (and landlords and employers and so on) were, in fact, using such discriminatory criteria. The current better practices aren't some natural result of banks somehow filling up with well-meaning people.

Re: Cathy O’Neil on Weapons of Math Destruction

#40

Post by spangry »

spangry wrote: post #8
theoh wrote: post #4

There's a fine paper called "Bias in Computer Systems" which is helpful in thinking about these problems. https://www.nyu.edu/projects/nissenbaum/papers/biasincompute... I found this recently in an appendix of Susan Leigh Star's book "Standards and their stories" which outlines a syllabus for the teaching of "infrastructure studies". The reading list also discusses the consequences of systems of categorization such a…

For well over a year now I've been mulling over about the damaging effects of DSM's categorisation of mental pathology. It had never occurred to me that this issue might be of more universal concern. For what it's worth, I think the damage caused by DSM stems from inaccurately modelling mental illness as: (1) binary propositions instead of assessing functionality/dysfunctional on a continuum (i.e. you either have 'ma…

Ah crud. That should actually be "Stahl's Essential Psychopharmacology".
Post reply on HN