Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

time.com

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#41

Post by sharemywin »

yummyfajitas wrote: post #33
sitkack wrote: post #28

If we train a mortgage engine on Redlining [0] data, it will then red line. Algorithms are not somehow "fair" because there isn't directly a human in the path. They are as "human as we are" but only proxy and at much lower fidelity. Not to Godwin the thread ... but big data has been used for immoral purposes since well, big data [1]. The even larger danger, is that people who don't know better, will simply "train a m…

This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid . I discuss this in detail here, with numerical examples: https://www.chrisstucchio.com/blog/2016/alien_intelligences_... The fact is that machine learning takes biased inputs and produces unbiased…

red-line - "The term refers to the presumed practice of mortgage lenders of drawing red lines around portions of a map to indicate areas or neighborhoods in which they do not want to make loans. Redlining on a racial basis has been held by the courts to be an illegal practice."

What's interesting is if at some point in future regulators extend "red line" to mean variable highly correlated to a protected class. Because if your using a variable and not checking how it correlates to a protected class and making financial based decisions on it. Couldn't you in affect transform the decision into a graph "women" on on side "men" on another(or what ever your favorite protected class is).

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#42

Post by yummyfajitas »

muraiki wrote: post #26
holografix wrote: post #23

I found this article problematic. First the assertion that "big data comes with all the biases of its creators". Isn't this exactly what big data (which I believe is a term used here to reference some sort of predictive model or machine learning) is supposed to avoid - ie. Making decisions based on evidence and not bias? Maybe it's the cynic in me but someone who worked in a hedge fund, started yet another advertisin…

Big data somehow being unbiased and impartial is a myth. The algorithms are still a product of humans. One notable example is a Google algorithm for labeling objects in photos that tagged black people as gorillas [1]. Now I don't have particular insight into the development process behind this algorithm, but this result may have occurred because the training data used did not include enough photos of black people. An…

What evidence do you have that one specific classification error resulting in a tweetstorm was the result of bias at all? I've been mislabeled as a tank in at least one image processing demo (admittedly, this was well before the era of deep learning). Image classification errors happen.

Also, it's a bit strange that you suggest ML algos are racist against Asians. You do know that Asians are wildly overrepresented among people building such algos, right?

Consider the possibility that some classification problems are actually more difficult than others. As one example, darker objects are harder to distinguish than bright ones and there are very clear mathematical reasons for that. That's true even in fields where questions of "racism" don't make sense, e.g. in MRI (magnetic resonance imaging is non-optical, but contrast still matters): https://www.chrisstucchio.com/pubs/wf_segment.pdf

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#43

Post by pzh »

I've always taken it for granted that discrimination and inequality would be the natural result of any optimization/ML algorithm, especially when maximizing for profit or any other utilitarian goal. I wonder why people would be surprised by this logical outcome given that it has been long known that statistical discrimination is profitable, and companies rarely have incentives to improve equality and diversity.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#45

Post by guelo »

bko wrote: post #40

> [algorithms] decide who gets access to credit and who pays higher insurance premiums, as well as who will receive online advertising for luxury handbags versus who’ll be targeted by predatory ads for for-profit universities. As opposed to what? Subjective decisions made by humans based on biases and personal preferences with no semblance of reason or fairness? > for example, who lives in an area targeted by crime f…

> the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?

If we were to enforce all laws we would need massively more police in every neighborhood. When police are sent to flood a poor neighborhood they end up ticketing poor residents for a bunch of "violations" that they would also find if they went to the rich neighborhoods. But if they ever actually did issue all those ticky-tack tickets in the rich neighborhoods residents would be up in arms contacting their city council members and it would end quickly. In the poor neighborhoods nobody ever hears about it and the residents that can least afford it take the financial hits.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#46

Post by yummyfajitas »

SamBam wrote: post #39
yummyfajitas wrote: post #33

Earlier quoted context omitted.

This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid . I discuss this in detail here, with numerical examples: https://www.chrisstucchio.com/blog/2016/alien_intelligences_... The fact is that machine learning takes biased inputs and produces unbiased…

> The fact is that machine learning takes biased inputs and produces unbiased outputs That is not necessarily true, because it assumes that the biased inputs have no influence on the outputs. Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often. Cause…

Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often.

But if the relationship is actually `defaultProb = a x interest_rate + b`, then the algorithm will reflect that. I discuss this case explicitly in the blog post I linked to in the section "What if black people don't perform as well?", and explicitly give an example of a linear model NOT doing what you claim it will do.

The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other.

No, it wouldn't because you'd also have cases of white people with high interest who defaulted. The model using race as a predictive variable would fail to fit those data points. I explicitly discuss this case in my blog post.

You are correct that in some cases you don't have enough training data. But in these cases, there is no particular reason for bias to have a particular sign. Again, as I discuss in the blog post I linked, why would Captain Kirk be biased against "black on left guy" vs "black on right guy"?

Insufficient training data causes variance, not bias, and could just as easily result in bias in favor of blacks. (In fact, I link to a number of examples where it does.)

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#47

Post by lmm »

yummyfajitas wrote: post #38
lmm wrote: post #36

Earlier quoted context omitted.

> This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid. If you train the algorithm to make the same decisions humans did (which is reasonably common when the humans are considered to be effective experts), it will make the same decisions humans did. I…

But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function. I algorithmically trade the stock market. I don't train my model by asking whether it trades the way I would. The whole point is for it to do a better job than me! I train the model to maximize my returns [1] in backtests and simulations - if it trades differently than I do, so much the better! [1…

> But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function.

In the idea case you would, sure. In practice mortgages take 25 years to deliver outcome data and the point isn't necessarily to get better outcomes, the mortgage provider would be content to get exactly the same outcomes (or even slightly worse outcomes) if it let them replace a large number of human employees with an automated system.

In most fields of human endeavour you can't backtest, because you don't have access to the outcomes of the decisions you didn't make. And the data often aren't as clear-cut or objective as market prices. The stock market is great but it's atypical of modern "big data" in many ways.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#48

Post by hood_syntax »

bko wrote: post #40

> [algorithms] decide who gets access to credit and who pays higher insurance premiums, as well as who will receive online advertising for luxury handbags versus who’ll be targeted by predatory ads for for-profit universities. As opposed to what? Subjective decisions made by humans based on biases and personal preferences with no semblance of reason or fairness? > for example, who lives in an area targeted by crime f…

> So having more cops in a high-crime neighborhood is somehow a bad thing, because the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?

Did you understand what you quoted? The author is saying that citations for petty offences (which practically everyone commits with some non-zero frequency, intentionally or not) have a higher chance of affecting future prospects of people in communities where there are more police. the Author isn't saying police are bad in general, rather that the negative effects of police presence are more significant in communities with higher police presence.

> So targeted advertising to the needs of people is bad because... I can't even make sense of this one.

The message isn't that hard to get, come on. The author is saying (assuming you think like them) that ignorance of suffering is a bad thing. You can say something like "why should I care about people I don't know", and I would understand and even agree to a large part with that statement, but this isn't James Joyce here, you can make sense of it.

Re: Mathematician Says Big Data Is Causing a ‘Silent Financial Crisis’

#49

Post by muraiki »

yummyfajitas wrote: post #42
muraiki wrote: post #26

Earlier quoted context omitted.

Big data somehow being unbiased and impartial is a myth. The algorithms are still a product of humans. One notable example is a Google algorithm for labeling objects in photos that tagged black people as gorillas [1]. Now I don't have particular insight into the development process behind this algorithm, but this result may have occurred because the training data used did not include enough photos of black people. An…

What evidence do you have that one specific classification error resulting in a tweetstorm was the result of bias at all? I've been mislabeled as a tank in at least one image processing demo (admittedly, this was well before the era of deep learning). Image classification errors happen. Also, it's a bit strange that you suggest ML algos are racist against Asians. You do know that Asians are wildly overrepresented amo…

I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning).

As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be.

Another example I heard recently from a conference was that of analyzing blighted homes in areas. The underlying data was generated by human inspectors who turned out to have given passes to violations in more affluent areas and fines in less affluent areas. The fines compounded the problem of blight as the residents could not afford to pay them and ended up causing a feedback loop of blight increasing in neighborhoods identified as blighted.

I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. I apologize if I have only added confusion rather than a productive analysis.

Post reply on HN