Cathy O’Neil on Weapons of Math Destruction
econtalk.org
Cathy O’Neil on Weapons of Math Destruction
1–10 of 42 posts
Re: Cathy O’Neil on Weapons of Math Destruction
#2This is an interesting read that is related. I wonder if she covers this specific topic in her book in her book https://www.propublica.org/article/machine-bias-risk-assessm.... Machine bias in prison sentencing.
Re: Cathy O’Neil on Weapons of Math Destruction
#3Ah, so finally someone else has caught on to the "Implements of Maths Instruction" joke we had back in 1999.
Re: Cathy O’Neil on Weapons of Math Destruction
#4There's a fine paper called "Bias in Computer Systems" which is helpful in thinking about these problems.
https://www.nyu.edu/projects/nissenbaum/papers/biasincompute...
I found this recently in an appendix of Susan Leigh Star's book "Standards and their stories" which outlines a syllabus for the teaching of "infrastructure studies". The reading list also discusses the consequences of systems of categorization such as the DSM and medical notions of gender.
Re: Cathy O’Neil on Weapons of Math Destruction
#5This is an interesting read that is related. I wonder if she covers this specific topic in her book in her book https://www.propublica.org/article/machine-bias-risk-assessm... . Machine bias in prison sentencing.
Yes, and it's a large part of the conversation in this interview.
Re: Cathy O’Neil on Weapons of Math Destruction
#6Her PR team seems to have been doing a great job. Can't go more than a day or two without seeing something about this book.
Re: Cathy O’Neil on Weapons of Math Destruction
#7Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc.
In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour, and the ability of the bank to consistently make better judgements means they can also offer lower rates and fees to people they choose to extend credit to.
This is in contrast to sentencing, where you prefer (in theory, at least) to bias judgements in favour of leniency (in the spirit of Blackstone's formulation), and you might even to prefer to make mistakes, if making the right decisions alienates people who interact with the justice system.
Re: Cathy O’Neil on Weapons of Math Destruction
#8There's a fine paper called "Bias in Computer Systems" which is helpful in thinking about these problems. https://www.nyu.edu/projects/nissenbaum/papers/biasincompute... I found this recently in an appendix of Susan Leigh Star's book "Standards and their stories" which outlines a syllabus for the teaching of "infrastructure studies". The reading list also discusses the consequences of systems of categorization such a…
For well over a year now I've been mulling over about the damaging effects of DSM's categorisation of mental pathology. It had never occurred to me that this issue might be of more universal concern. For what it's worth, I think the damage caused by DSM stems from inaccurately modelling mental illness as:
(1) binary propositions instead of assessing functionality/dysfunctional on a continuum (i.e. you either have 'major depressive disorder' or you don't); and
(2) discrete and distinct 'diseases', instead of the cumulative effects of dysfunction/abnormality in multiple neural 'sub-systems'.
If you're interested in the DSM case, and the likely way forward (from my understanding, the convergence of psychiatry and neurobiology and increasingly accurate and affordable neuro-imaging techniques), the textbook "Stahl's Essential Pharmacology" is well worth a read.
Re: Cathy O’Neil on Weapons of Math Destruction
#9Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…
> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour
I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on their payments. There are a lot of factors that go into the formula to calculate that risk today, especially things like previous payment patterns and previous borrowing history. There are some factors that do NOT go into the formula like the borrower's race or the payment patterns of family members.
Now, I am sure (although I have not actually run the numbers) that an analysis would show that race and family-member credit scores are fairly strongly correlated with default rates. That means that a credit card company chose to use factors like race as part of the scoring decision the company would do better than another bank that didn't use those factors.
But we do not WANT to be using race or family wealth to decide credit decisions. Speaking as a banker, my company does not want to be using those criteria; speaking as a citizen, my country does not want banks to be using those criteria. Restricting what criteria are permissible for making credit decisions enables the banks that would refuse to use that data for ethical reasons to remain competitive. Restricting it allows us to craft a society where any citizen has an equal chance of success.... well, that may be a stretch but at least it is CLOSER to such a society than if we did not have such restrictions.
Sure, you can categorize this as "erroneous decisions in the favor of those who come from a poor or minority background", and I suppose you are technically correct. But that pretty phrasing doesn't make it ethical.
Re: Cathy O’Neil on Weapons of Math Destruction
#10Cathy's use of 'problematic' seems to paper over any need to actually go into an analysis of cost and benefits of using statistics and statistical learning in sentencing, personal banking etc. In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in…
> In the business of making credit decisions (specifically, who to extend credit to, and at what rates), preventing banks from using better information only harms people who would have been the subjects of erroneous decisions in their favour I disagree. For instance, in the credit-card business a key factor in the ability of a bank to be successful is the ability to assess the likelihood of a person defaulting on the…
This is well put, and speaks to me of why we do need to legislate morality at times.