A Way to Detect Bias • Hacker News Board

Re: A Way to Detect Bias

#71

Post by ChicagoBoy11 » Sat, Oct 31, 2015, 9:59 PM UTC

WildUtah wrote: post #61

ChicagoBoy11 wrote: post #52

Earlier quoted context omitted.

Great insight. I had the great fortune to take a course with Gary Becker where we went into some of the mathematics of college admissions. He made this precise point -- the variances of the particular populations you are looking at matter a great detail. He managed to build some pretty convincing models which provided a compelling narrative for "biases" that we seemed to observe in the real world, all with simple cha…

I had the great fortune to take a course with Gary Becker Lucky you. He was great. I never had the chance to take a class from him but it would have been worth a Chicago winter to have the chance.

Second-best: https://www.youtube.com/watch?v=QajILZ3S2RE&list=PL9334868E7...

(I took this very class the year prior)

Re: A Way to Detect Bias

#72

Post by WildUtah » Sat, Oct 31, 2015, 10:01 PM UTC

rewqfdsa wrote: post #43

WildUtah wrote: post #20

On a simple mathematical basis, this is false. Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80% or better chance of graduation. Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%, with the same average but less variation i…

Why are you using biased mathematics? If statistics and the scientific method (the tools through which dead white men continue to colonize) give us obviously problematic results, we should abandon them in favor of a method of inquiry that promotes social justice.

I enjoyed this comment.

Re: A Way to Detect Bias

#73

Post by eoi » Sat, Oct 31, 2015, 10:02 PM UTC

WildUtah wrote: post #20

On a simple mathematical basis, this is false. Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80% or better chance of graduation. Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%, with the same average but less variation i…

The article defines bias as follows:

> Want to know if the selection process was biased against some type of applicant? Check whether they outperform the others. This is not just a heuristic for detecting bias. It's what bias means.

Under that definition, you have been biased against A. [edit: on reflection I see this as a weakness of his definition. I missed that your selection process does in fact select the best candidates.]

Re: A Way to Detect Bias

#74

Post by d0m » Sat, Oct 31, 2015, 10:03 PM UTC

I don't get it :-/ Why is there a bias?

Even if the VCs are totally unbiased, why couldn't the startups with women outperformed the others? It could happen for a variety of reasons. Just hypothetically speaking, maybe startups-with-women have different networking connections or insight that male-only-startups don't have?

Re: A Way to Detect Bias

#75

Post by quietplatypus » Sat, Oct 31, 2015, 10:05 PM UTC

WildUtah wrote: post #20

On a simple mathematical basis, this is false. Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80% or better chance of graduation. Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%, with the same average but less variation i…

Very interesting! Seems like I don't understand probability as well as I think I do, since I bought the argument till I ran into this comment.

So, what are the effects of variance in different evaluation contexts, and do we have a meaningful way to measure bias if we take variance into account?

My initial reactions:

- It seems the higher the variance, the better examples you can trot out to say you are not biased against that particular group. since youll always find a member of that group who does amazing.

- If distributions of performance are multimodal, its even harder to conclude stuff because different institutions might cut off different modes when selecting the bar.

- Modeling the sources of variance may lead to insight into any actual bias.

Re: A Way to Detect Bias

#76

Post by Alex3917 » Sat, Oct 31, 2015, 10:08 PM UTC

d0m wrote: post #74

I don't get it :-/ Why is there a bias? Even if the VCs are totally unbiased, why couldn't the startups with women outperformed the others? It could happen for a variety of reasons. Just hypothetically speaking, maybe startups-with-women have different networking connections or insight that male-only-startups don't have?

> Why is there a bias?

Applications by definition are supposed to be biased. If an application weren't biased then it wouldn't be an application, it would just be a lottery. And while that's a system that a lot of charter schools actually use, or a least pretend to, I think it would be a tough sell to convince venture capitalists to allocate their capital this way.

Re: A Way to Detect Bias

#77

Post by [deleted] » Sat, Oct 31, 2015, 10:09 PM UTC

mattmaroon wrote: post #59

WildUtah wrote: post #20

On a simple mathematical basis, this is false. Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80% or better chance of graduation. Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%, with the same average but less variation i…

So in this particular case, assuming that First Round's sample size is significant, it may just be that the female founders who seek them out are just on average better than the male ones? I suppose that if women think that the selection process is biased against them, and most do (and it may be) perhaps the less than excellent ones just don't apply, whereas that isn't true for males?

[deleted]

Re: A Way to Detect Bias

#78

Post by tprice7 » Sat, Oct 31, 2015, 10:09 PM UTC

yummyfajitas wrote: post #41

MattHeard wrote: post #37

Earlier quoted context omitted.

Wouldn't the weakest male and weakest female founders be underperforming more due to individual factors than bias? Maybe a mean of the lower quartile would remove some the the noise.

Suppose the cutoff for men is C but for women is C+K. Then the weakest man can be expected to have quality C+epsilon, while the weakest woman will have quality C+K+epsilon. Here epsilon is how close a typical candidate will be to the cutoff, and is mainly a function of the sample size. I don't know the behavior of epsilon off the top of my head.

I think MattHeard understood this and his point was that the weakest man will realistically have quality C + epsilon + random, and the weakest woman will have quality C + K + epsilon + random. The random term arises because no evaluation process is going to perfectly tell you how people are going to end up performing. But yeah, this seems fixable also, by averaging some number of the lowest performing members of each group, as MattHeard suggested.

Re: A Way to Detect Bias

#79

Post by tansey » Sat, Oct 31, 2015, 10:11 PM UTC

yummyfajitas wrote: post #66

tansey wrote: post #64

Earlier quoted context omitted.

Does that assume both samples are identically distributed and the only difference is the cutoff? If it does, then couldn't we just continue to do a difference of means test and still be consistent? If it doesn't, how do you handle identifying the cutoff minima and the two different distributions in a frequentist way?

The only assumption I need is that P_{f,g}([C,C+d]) >= h(d) > 0 for some arbitrary monotonic function h(d). This comes directly from the p-value formula. I.e., for any d, there is a finite probability of finding an A or a B in [C,C+d]. I don't actually care what the shapes of f or g are at all beyond this - as long as this probability exists and is bounded below (in whatever class of functions f and g might be drawn…

Sorry, I'm confused here. A p-value makes an implicit assumption that your null hypothesis is a known N(0,1). That may be throwing me off a bit. I get the point of you want to look at the likelihood function which is just one minus the CDF in the given interval. I'm just not clear on how you can get around f and g being arbitrarily parameterized functions of a given class. Are you assuming we know the class and something about f?

Re: A Way to Detect Bias

#80

Post by tprice7 » Sat, Oct 31, 2015, 10:13 PM UTC

SFLemonade wrote: post #53

rewqfdsa wrote: post #43

Earlier quoted context omitted.

Why are you using biased mathematics? If statistics and the scientific method (the tools through which dead white men continue to colonize) give us obviously problematic results, we should abandon them in favor of a method of inquiry that promotes social justice.

Please tell me this is sarcasm...

Based on rewqfdsa's comment history, this is probably sarcasm.