The Mathematics of Paul Graham's Bias Test • Hacker News Board

Re: The Mathematics of Paul Graham's Bias Test

#31

Post by agarden » Fri, Nov 06, 2015, 6:46 AM UTC

lukev wrote: post #10

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performa…

Let's assume that bias does persist past selection through the duration of the program. Does that change the interpretation when you look at First Round Capital's data that shows its female founders outperforming the males by 63%? I don't think it does.

The test may not be sufficient to prove that you have no bias, but it may be good enough to prove that you do. When it does indicate bias, it seems likely to be correct.

To put it another way, if it is 1948 and the only three black people in Major League Baseball are all superstars, then the distribution of baseball skill among black players is extremely unbalanced or there is a lot of bias keeping the average and moderately-better-than-average black players out.

Re: The Mathematics of Paul Graham's Bias Test

#32

Post by [deleted] » Fri, Nov 06, 2015, 6:48 AM UTC

ceph_ wrote: post #24

bsder wrote: post #13

The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place. "But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason. If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.

Dropping outliers is common in statistical analysis.

[deleted]

Re: The Mathematics of Paul Graham's Bias Test

#33

Post by nkurz » Fri, Nov 06, 2015, 6:50 AM UTC

escape_goat wrote: post #16

danieltillett wrote: post #12

Earlier quoted context omitted.

This is really important. Lets say the whole VC ecosystem is biased against redheads (just to pick a random group). What would happen is the redheads would under perform other groups as they were discriminated against at each stage of the VC lifecycle. They would not show up as a group that over performing later. The only bias you can detect using Paul’s approach is bias that only applies at the initial stage and not…

Just to cross the beams of pedantry here for a moment, a widespread and well known --- if less than serious or systemic --- cultural/social bias against red haired people, probably first coming into public consciousness in North America due to the infamous South Park 'Ginger' episode, has in fact primed you to select "redheads" as a non-contentious example of a plausibly ethnic group that might be discriminated again…

I'd prefer to imagine that it's because the majority of the HN readership reads Nature for their regular dose of science fiction: http://www.nature.com/nature/journal/v453/n7194/full/453562a...

Alternatively, I'd be OK imagining that it was subconsciously chosen here not at random, but because I used this reference for an example in the previous thread.

Is South Park another journal worth reading? Are they open access?

Re: The Mathematics of Paul Graham's Bias Test

#34

Post by shoyer » Fri, Nov 06, 2015, 6:57 AM UTC

I think the more fundamental flaw in PG's argument, which is just as present here, is that it assumes the populations are otherwise identical. That's obviously not the case -- there's no random assignment for bias -- so this sort of test can't tell you anything direct about casuation.

Any credible statistical test for bias should be framed in the language of causal inference, e.g., as described by Judea Pearl: http://ftp.cs.ucla.edu/pub/stat_ser/r350.pdf

Re: The Mathematics of Paul Graham's Bias Test

#35

Post by LordHumungous » Fri, Nov 06, 2015, 7:07 AM UTC

zodiac wrote: post #25

LordHumungous wrote: post #19

Earlier quoted context omitted.

Yes I get that. The crux of his argument is: >Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math.

> His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math. Not really, his argument is that "PG's mean-post selection test (the 'PMST') is only valid if different sexes have the same distribution of abilities". If you or PG believe that the PMST is a valid way of showing that bias exists, the bu…

From the Paul Graham article:

>You can use this technique whenever (a) you have at least a random sample of the applicants that were selected, (b) their subsequent performance is measured, and (c) the groups of applicants you're comparing have roughly equal distribution of ability.

So yes, OP is ignoring the entire premise of PG's argument.

Re: The Mathematics of Paul Graham's Bias Test

#36

Post by jamiequint » Fri, Nov 06, 2015, 7:21 AM UTC

ceph_ wrote: post #24

bsder wrote: post #13

The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place. "But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason. If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.

Dropping outliers is common in statistical analysis.

Dropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' entire goal is to find outliers.

Re: The Mathematics of Paul Graham's Bias Test

#37

Post by zodiac » Fri, Nov 06, 2015, 7:29 AM UTC

LordHumungous wrote: post #35

zodiac wrote: post #25

Earlier quoted context omitted.

> His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math. Not really, his argument is that "PG's mean-post selection test (the 'PMST') is only valid if different sexes have the same distribution of abilities". If you or PG believe that the PMST is a valid way of showing that bias exists, the bu…

From the Paul Graham article: >You can use this technique whenever (a) you have at least a random sample of the applicants that were selected, (b) their subsequent performance is measured, and (c) the groups of applicants you're comparing have roughly equal distribution of ability. So yes, OP is ignoring the entire premise of PG's argument.

OP acknowledges this... "Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C."

To me the rest of the article is asks the question, "requirement (c) is really strong, is there a way we can use post-selection statistics to determine bias while weakening (c)? what if we tried measuring the post-selection minimum instead of the mean?"

Also PG edited his essay to add that disclaimer only after WildUtah's comment, so it's possible that OP hasn't read the updated version.

Re: The Mathematics of Paul Graham's Bias Test

#38

Post by angelbob » Fri, Nov 06, 2015, 7:37 AM UTC

haberman wrote: post #28

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…

Yup. The author mentions that noise is a serious problem for this method in the article, and talks about some ways he looked at trying to reduce it, but didn't come up with a good one.

Re: The Mathematics of Paul Graham's Bias Test

#39

Post by slugg » Fri, Nov 06, 2015, 7:38 AM UTC

jamiequint wrote: post #36

ceph_ wrote: post #24

Earlier quoted context omitted.

Dropping outliers is common in statistical analysis.

Dropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' entire goal is to find outliers.

Don't just throw around some Peter Thiel shit like it justifies any argument you want it to.

Re: The Mathematics of Paul Graham's Bias Test

#40

Post by angelbob » Fri, Nov 06, 2015, 7:39 AM UTC

jamiequint wrote: post #36

ceph_ wrote: post #24

Earlier quoted context omitted.

Dropping outliers is common in statistical analysis.

Dropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' entire goal is to find outliers.

Possible. In this case, we're not looking for outliers or measuring based on financial success, but trying to tell if the VC is systematically biased anti-woman.

It's not clear that dropping outliers is a bad idea there. It's also not clear it's a good idea, granted.