Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

time.com

Post reply on HN
Print view View original

11–20 of 111 posts

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#11

Post by nl »

jchrisa wrote: post #3

Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Here's the last paragraph of the article: """ O’Neil also proposes updating existing civic rights oriented legislation to make it clear that it encompasses computerized algorithms. One thing that her book has already made quite clear – far from being coolly scientific, Big Data comes with all the biases of i…

Disallowing the use of race as a signal in machine learning

That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient.

See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover):

  Feature6578 = Loc=EastOakland && Income
I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient.

Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder.

[1] http://deliprao.com/archives/129

[2] https://www.chrisstucchio.com/blog/2016/alien_intelligences_...

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#12

Post by ThePhysicist »

There is a lot of interesting research on the societal impact and the (potential) problems caused by the use of large-scale data analysis and artificial intelligence, e.g. by Kate Crawford at MSFT Research.

Last year I gave a talk on this at 32C3 where I tried to elucidate some of the problems that can pop up when using data analysis to automate things that were previously done by humans:

https://www.youtube.com/watch?v=iRY9IceaVig

Personally I see a lot of potential in data analysis & artificial intelligence, but like all technologies they pose significant risks as well. What we really need therefore (IMHO) is to teach ethics to people that work with data, and establish organizations and methods that ensure data analysis isn't used to do harm (deliberately or not).

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#13

Post by Noseshine »

ThePhysicist wrote: post #12

There is a lot of interesting research on the societal impact and the (potential) problems caused by the use of large-scale data analysis and artificial intelligence, e.g. by Kate Crawford at MSFT Research. Last year I gave a talk on this at 32C3 where I tried to elucidate some of the problems that can pop up when using data analysis to automate things that were previously done by humans: https://www.youtube.com/watc…

    > What we really need therefore (IMHO) is to teach ethics to people
The issue with almost all organized teaching (as opposed to when you don't even know you are doing it, e.g. to your children, every day) is that it only targets conscious thinking. Do you already see the problem with teaching "ethics" (and empathy)? It's the wrong brain area. People will ace the tests but how they actually act won't change. Another big reason for why teaching ethics (the way it is usually done) is that much of behavior is environment-driven. So you can teach someone all you want, if you then place them in a cut-throat competitive "result-driven" environment you can see empathy and ethics quietly sneak out the back door.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#14

Post by denzil_correa »

jchrisa wrote: post #3

Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Here's the last paragraph of the article: """ O’Neil also proposes updating existing civic rights oriented legislation to make it clear that it encompasses computerized algorithms. One thing that her book has already made quite clear – far from being coolly scientific, Big Data comes with all the biases of i…

> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness

Many a time, algorithms infer race from secondary or tertiary variables. This problem is compounded by two more factors - (1) Unable to explain WHY an algorithm has reached a decision aka interpretability and (2) Focus of algorithm building to maximize accuracy regardless of other valid costs. There is some pretty interesting work going on this area.

http://fairness.haverford.edu/

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#15

Post by s_q_b »

Chris2048 wrote: post #9

> add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation He'll also be less likely to be the victim of violent crime in an area with a proven record for violent crime. For the price of avoiding 'petty' crimes, it seems like a good trade-off.

That's a solid theoretical argument. However, it is contradicted by the empirical evidence.

What usually happens is that the police issue fines so frequently that they become considered an occupying force by the populace. In turn, the populace withholds information from the police, and both violent crimes and petty violations rise.

Police statistics make quantitative conclusions difficult, since they are so often fudged for political purposes. They rise when the police need more funding, and fall when they need political capital. Murder rates are generally considered the only reasonably accurate figures.

On the ground, however, the reality is very clear.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#16

Post by skywhopper »

ThePhysicist wrote: post #12

There is a lot of interesting research on the societal impact and the (potential) problems caused by the use of large-scale data analysis and artificial intelligence, e.g. by Kate Crawford at MSFT Research. Last year I gave a talk on this at 32C3 where I tried to elucidate some of the problems that can pop up when using data analysis to automate things that were previously done by humans: https://www.youtube.com/watc…

People learning or not learning ethics doesn't really have much to do with it. Read up on the Fraud Triangle. Unethical job tasks follow a similar pattern. Perceived need ("I might lose my job if I refuse") and rationalizations ("the results of this flawed algorithm surely won't get taken seriously") get you two-thirds of the way there. The third leg, opportunity, is inherent in this sort of job. So "ethics" are really beside the point when the workers confront the reality of their situation.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#17

Post by mseebach »

nl wrote: post #11
jchrisa wrote: post #3

Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Here's the last paragraph of the article: """ O’Neil also proposes updating existing civic rights oriented legislation to make it clear that it encompasses computerized algorithms. One thing that her book has already made quite clear – far from being coolly scientific, Big Data comes with all the biases of i…

Disallowing the use of race as a signal in machine learning That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient. See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover): Feature6578 = Loc=EastOakland && Income I don't know what th…

The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but so is a white person with those characteristics (and a black person in Connecticut making $250k is going to do fine and so is a white person). But if more black people than white legitimately are in a situation where they might struggle to service a mortgage, more black people than white would get flagged as such by an objective algorithm. Does that make the algorithm racist? The slide seems to imply so (although without speaker-notes it's difficult to ascertain).

The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#18

Post by nickflees »

s_q_b wrote: post #15
Chris2048 wrote: post #9

> add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation He'll also be less likely to be the victim of violent crime in an area with a proven record for violent crime. For the price of avoiding 'petty' crimes, it seems like a good trade-off.

That's a solid theoretical argument. However, it is contradicted by the empirical evidence. What usually happens is that the police issue fines so frequently that they become considered an occupying force by the populace. In turn, the populace withholds information from the police, and both violent crimes and petty violations rise. Police statistics make quantitative conclusions difficult, since they are so often fud…

Interesting. Do you have a source?

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#19

Post by mseebach »

s_q_b wrote: post #15
Chris2048 wrote: post #9

> add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation He'll also be less likely to be the victim of violent crime in an area with a proven record for violent crime. For the price of avoiding 'petty' crimes, it seems like a good trade-off.

That's a solid theoretical argument. However, it is contradicted by the empirical evidence. What usually happens is that the police issue fines so frequently that they become considered an occupying force by the populace. In turn, the populace withholds information from the police, and both violent crimes and petty violations rise. Police statistics make quantitative conclusions difficult, since they are so often fud…

In a world where we have police, we want to deploy it against crime. If the police isn't fit for purpose, that's a different kettle of fish.

(Also, the next bit of the quote, "Yet neighborhoods more likely to commit white collar crime aren’t targeted in this way", suggests a very loose grasp of how policing works. No, they're not, because patrolling the street in front of an office building isn't the least bit effective against white collar crime.)

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#20

Post by nl »

mseebach wrote: post #17
nl wrote: post #11

Earlier quoted context omitted.

Disallowing the use of race as a signal in machine learning That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient. See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover): Feature6578 = Loc=EastOakland && Income I don't know what th…

The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but…

There isn't a simple answer here - like in most non-trivial data science/ML questions.

The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal.

The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?

Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).

Post reply on HN