Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Way to Detect Bias

paulgraham.com

Post reply on HN
Print view View original

21–30 of 224 posts

Re: A Way to Detect Bias

#21

Post by srtjstjsj »

devalier wrote: post #14

Fortunately there's a way to measure bias that's much more reliable, when it can be used....A couple months ago, one VC firm (almost certainly unintentionally) published a study showing bias of this type. First Round Capital found that among its portfolio companies, startups with female founders outperformed those without by 63%. Except if you want to use statistics to measure bias, you need a statistically significa…

> Does he honestly believe that First Round is biased against founders from elite colleges?

Maybe FR is. Imagine that elite college is highly predictive of success, so you prefer to pick elite college grads, all other available evidence being equal. You're biased toward elite college grads, right?

But what if elite college grads really are phenomenally more successful, and you can't see the detailed reason (high school experience, network, whatever), to the point that they are all better than all non-elite college grads. Then even selecting 90% of your pool from elites, and 10% from the rest, is biased against the actual merit of the applicants.

[these numbers are totally made up. I'm not saying elite college grads really have these characteristics.]

The trickiness is that you can't see everything when you evaluate, so you have to assign weights to the factors you have, and leverage corellations to hidden important factor.

Re: A Way to Detect Bias

#22

Post by pbnjay »

Fittingly, another type of bias observed in the linked report is that against Solo Founders. The report states that solo founders do worse when measured against the same yardstick as multiple founders. Maybe from a VC perspective this is intended (big raise => bigger exit?), but I'd argue that you don't need to raise as much when you have a solo founder because dilution is less of a concern.

Re: A Way to Detect Bias

#23

Post by pbhjpbhj »

aceperry wrote: post #12
newjersey wrote: post #6

Earlier quoted context omitted.

If students of Asian origin outperform the whole student body, can we conclude admissions folks are biased against students of Asian origin?

That's possible. Without concrete data to give you, there are some suspicions that because of the much better performance of Asian students, they're being limited in the admissions process. Otherwise, Asian students would make up the vast majority of the students admitted. This would crowd out the non-Asian students accepted for admission. In this case, it would be more accurate to say that admissions officers limit…

>admissions officers limit the number of Asian students accepted instead of saying admissions folks are biased against Asian students //

The effect is the same isn't it? Less chance for a student with ancestors from a particular geographic locale getting a placement.

Re: A Way to Detect Bias

#24

Post by tansey »

devalier wrote: post #14

Fortunately there's a way to measure bias that's much more reliable, when it can be used....A couple months ago, one VC firm (almost certainly unintentionally) published a study showing bias of this type. First Round Capital found that among its portfolio companies, startups with female founders outperformed those without by 63%. Except if you want to use statistics to measure bias, you need a statistically significa…

So true.

PG's articles are generally filled with good intuitive insight. Unfortunately, statistics can be very tricky to turn into folksy wisdom. Rules of thumb like "you need 30 samples before you can say anything" that are derived from the CLT are a good example of ones that work well enough in practice, even if they obscure some underlying subtleties. This article is an example of a rule that sounds simple, but actually has so many asterisks that one would expect it to be mostly useless in practice.

If women are performing better on average, it doesn't mean that you should invest in more women necessarily. What if all the remaining candidates would have a negative mean return? If they included Uber and all of a sudden the women now underperform men, does that mean they're biased against men and they need to invest in less women?

There's just so many statistical fallacies at play here that it's a shame that Jessica, Sam, or Geoff didn't point out that maybe someone with a stats background should read the article first before publishing it.

Re: A Way to Detect Bias

#26

Post by srtjstjsj »

cookiecaper wrote: post #18
newjersey wrote: post #6

Earlier quoted context omitted.

If students of Asian origin outperform the whole student body, can we conclude admissions folks are biased against students of Asian origin?

Alternately, you could conclude that instead of "mediocre" Asians being excluded by bias, Asians have an external advantage that makes them perform better. Maybe it's cultural, since most Asians are taught a very strong work ethic and heavy emphasis is placed on formal schooling, succeeding, and fitting in. Maybe Asians are physically better adapted to that type of work, with brains that retain information more easil…

Even if Asians perform better for external reasons, the selection process should account for that before the selection is made, and the admitted class should be roughly equal performers, as a group. Unless Asians have a very lumpy shaped performance curve across the group

Re: A Way to Detect Bias

#27

Post by nartz »

Interesting thoughts. However, this argument is biased because it assumes that the performance of the applicants WHO WERE ACCEPTED is not biased by the selection process itself, and that the performance characteristics of the selected sample are representative of the performance characteristics of the total, which could be a weak assumption.

An attempt at translating to mathematics (feel free to correct me!):

X = event that person belongs to group x

Y = event that person belongs to group y

S = event that person is selected

W = event that person will perform like a 'winner'

for simplicity P(X) + P(Y) = 1

Naturally, 'unbiased' in this case is simply P(S|X) = P(S1), and P(S|Y) = P(S2), i.e. that the selection process is independent of a certain variable X or Y

PG says we can measure the the performance of these selected applicant winners for each class, i.e. P(X|S,W).

I believe PG assumes that:

P(X|W) / P(Y|W) should equal P(X|S,W)/P(Y|S,W). We can see that these are different distributions, since the second is already conditioned on the selection process.

Simplified, PG assumes that P(X|S,W) = P(X|W) i.e. that conditioning on the selection process does not bias the winning results.

Its left for the reader exercise to determine the 'pathological' cases where this selection variable's distribution makes PG's assumption correct or incorrect.

However, this is simply theoretical - the actual distribution may or may not be 'pathological' and the assumptions made by PG could very well be good.

Re: A Way to Detect Bias

#28

Post by yummyfajitas »

cookiecaper wrote: post #18
newjersey wrote: post #6

Earlier quoted context omitted.

If students of Asian origin outperform the whole student body, can we conclude admissions folks are biased against students of Asian origin?

Alternately, you could conclude that instead of "mediocre" Asians being excluded by bias, Asians have an external advantage that makes them perform better. Maybe it's cultural, since most Asians are taught a very strong work ethic and heavy emphasis is placed on formal schooling, succeeding, and fitting in. Maybe Asians are physically better adapted to that type of work, with brains that retain information more easil…

That doesn't actually have a large effect. An example in numpy:

    In [14]: x = norm(0.0,1).rvs(100000)
    In [15]: mean(x[where(x > 2.0)])
    Out[15]: 2.3774795090391301
    In [16]: y = norm(0.5,1).rvs(100000)
    In [17]: mean(y[where(y > 2.0)])
    Out[17]: 2.4372124830289557
I.e., a difference in the mean of 0.5 sigma corresponds to 0.06 in Graham's test statistic.

Graham is a little bit off - a better place to look for bias is the bottom of the accepted distribution than at the mean.

Re: A Way to Detect Bias

#30

Post by [deleted] »

gizmo wrote: post #7

A related observation (which I've been making for a long time) is that the absence of mediocre women in positions of power is strong evidence of bias. Men can succeed when they're mediocre, but women have to be exceptional. Likewise for minorities.

[deleted]
Post reply on HN