Wow that's a really nicely rendered page.
TIL about https://www.mathjax.org/.
The Mathematics of Paul Graham's Bias Test
21–30 of 84 posts
Re: The Mathematics of Paul Graham's Bias Test
#22Earlier quoted context omitted.
Read it again. He's talking about the counterexample there. It's a hypothetical.
Yes I get that. The crux of his argument is: >Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math.
No, he's not asserting any such thing. Again, it's a hypothetical. A counterexample that means, yes, the "mean" test is flawed. Because it doesn't work in all scenarios.
Re: The Mathematics of Paul Graham's Bias Test
#23The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place. "But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason. If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.
This is why I love bootstrapping [1].
Re: The Mathematics of Paul Graham's Bias Test
#24The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place. "But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason. If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.
Dropping outliers is common in statistical analysis.
Re: The Mathematics of Paul Graham's Bias Test
#25Earlier quoted context omitted.
Read it again. He's talking about the counterexample there. It's a hypothetical.
Yes I get that. The crux of his argument is: >Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math.
> His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math.
Not really, his argument is that "PG's mean-post selection test (the 'PMST') is only valid if different sexes have the same distribution of abilities". If you or PG believe that the PMST is a valid way of showing that bias exists, the burden is on you to show that different sexes have the same distribution of abilities.
Re: The Mathematics of Paul Graham's Bias Test
#26> The idea is generally correct - bias in a decision process will be visible in post-decision distributions I find what's wrong with the idea more fundamental, that it talks only about the 'selection process' but in fact bias that impacts success or failure can come at other points.
This is really important. Lets say the whole VC ecosystem is biased against redheads (just to pick a random group). What would happen is the redheads would under perform other groups as they were discriminated against at each stage of the VC lifecycle. They would not show up as a group that over performing later. The only bias you can detect using Paul’s approach is bias that only applies at the initial stage and not…
It will be visible for the last biased actor. So begin testing the later rounds and work backwards.
Re: The Mathematics of Paul Graham's Bias Test
#27(comment reposted from the earlier submission that didn't catch: https://news.ycombinator.com/item?id=10513574)
Hi Chris ---
In the earlier thread, it seemed like some people were reaching different conclusions because they were using different definitions of "bias". I think my working definition would be something like "there existed in the actual applicant pool a subset of unfunded female founders who should have been statistically expected (given the information information available to the VC's at the time of decision) to outperform an equal sized subset of male founders who did in fact receive funding".
Alternatively (and I don't think equivalently?) one could reasonably take bias to mean "Given their prejudices, if the same VC's had been blinded to the sex of the applicants, they would have made funding choices resulting in higher total returns than the sex-aware choices they actually made." I'm sure there are many other ways of defining "bias". Could you define what would need to be true for your test to show that "the VC process is biased against female founders"?
Re: The Mathematics of Paul Graham's Bias Test
#28> So rather than comparing mean performance, we'll compare minimum performance.
If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test.
The minimum performer could be someone who had a sudden personal crisis. Or who had 10 competitors suddenly pop up. Or any number of other circumstances outside their control. The minimum performer is, almost by definition, an outlier. It doesn't seem rational to suppose that an outlier is representative of the group.
I can understand that statistically this test may be more rigorous. In practice I would expect it to be less rigorous. Because the assumption it makes (that a single outlier is representative of the group) seems even more dubious than the assumptions required for Paul's original idea.
Re: The Mathematics of Paul Graham's Bias Test
#29One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performa…
It's reasonable to think that the bias would decrease over the company's lifetime. In early rounds there is little data on the company, so more decisions are made on hunches, and there's a lot of potential influence for bias. While there's potential for that in later rounds too, there's also a lot more objective information. The company is either making money or it isn't.
The importance of relationships to the funding round also plays a role. If you get as far as an IPO, it seems unlikely that the stock-buying public is going to stay away because the founders are female/black/etc.
Re: The Mathematics of Paul Graham's Bias Test
#30I think this is a really good start for the most common types of bias. A few counter-examples that might slip through the cracks of this test:
Only examining the sample without looking at the population of applicants has its limits. Especially as multiple interviews becomes the norm, filters that don't affect the distribution of outcomes will be missed. For example, the person screening resumes might weed out anyone with an ethnic-sounding name. A different person, who is not biased, interviews the candidates. The quality of the candidates accepted will be the same, but the number of minority applicants will be smaller than it should be.
Measuring outcomes allows for external biases to distort the results. Start with a company that is biased against women, so that the average female founder is better than the average male. However, that same level of sexism exists in the market, such that the company's performance is hampered due to prejudice against the founder. The VC's bias would be hidden by the counter-bias in the market.