Interesting tests which can reveal your current implicit (read hidden) attitudes toward race, gender, color.
Worth to do and discover few facts about ourselves, even if uncomfortable.
The Mathematics of Paul Graham's Bias Test
71–80 of 84 posts
Re: The Mathematics of Paul Graham's Bias Test
#72Earlier quoted context omitted.
The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was u…
It's very interesting. And what are the most suitable formulas we can use to measure how robust it is?
If you are estimating a parameter with a sample statistic, two properties it should have are unbiasedness and low variance: https://en.wikipedia.org/wiki/Minimum-variance_unbiased_esti...
Re: The Mathematics of Paul Graham's Bias Test
#73Earlier quoted context omitted.
Why is it appropriate to drop outliers? (The fact that something is common does not make it a good thing.)
Statistics 101. When you have samples you throw away the highest and lowest member, to counteract some random occurrence. The mean net worth of the patrons in any restaurant carlos slim frequents rises substantially when he is there.
That, and the fact that outliers can often be discounted due to measurement/instrumentation error.
Moreover, the fact that Carlos entered your restaurant may be a significant event depending on the analysis that you're attempting to do. So you need to have to have a good rationale for dropping outliers, and you should probably also watch for bias when dropping outliers that don't support your hypothesis!
Re: The Mathematics of Paul Graham's Bias Test
#74Posting this, since no one seems to have pointed it out:
> So rather than comparing mean performance, we'll compare minimum performance.
1. This is a useless metric for startup investors to use, since (almost surely) the minimum performance in every group of reasonable size will 0 (the startup went out of business)... and this will be true even if the investor is biased.
2. The maximum statistic was rightly avoided here because for power-law distributed values (which startups returns are), you'd need to know the population sizes to estimate if the distribution of {A} was different than the distribution of {B}.
If you're willing to take on faith that both A and B have the same distribution, then the test is easy: is the acceptance rate for As significantly different than the acceptance rate for Bs? If you've invested in more than, say, 100 startups, you have a big enough sample to check this... this requires knowing the size of application pools, and who was accepted though.
3. I believe that in general it's not possible to determine a bias from the kind of aggregate statistic pg is discussing without at least some knowledge of the sample space.
For example, using OP's method, you will find that almost every selection process in the world is biased for you if you divide the world as {you} vs {non-yous} (you're doing significantly worse than the best non-you). And find that almost every selection process in the world is heavily biased against you if you use the minimum statistic (you're doing significantly better than the worst non-you). This is also true for smallish groups (eg {your friends} vs {not your friends}).
The same is true for PG's method -- it's highly unlikely that {you} fall exactly at the average value of {non-yous}, or that {your friends} fall exactly at the average of {not your friends}.
4. I believe that the math here is distracting from the core question.
Core question 1: Do men and women on average make the same choices?
If you believe that, then determining bias is easy: we already know who the investor funded. Is the number of men the investor funded different from the number of women? Yes? Then the investor is biased. This is much more direct than the the kind of forensic accounting pg is proposing.
I suspect that pg didn't propose this test because pg doesn't believe that men and women on average make the same choices. He knows, for example, that the number of female applicants to YC is different than the number of male applicants (a gendered difference in behavior). Google "men and women career choices" or similar if you're interested in learning more, or better yet, read some first person accounts from FTM men about the cognitive effects of taking testosterone.
Since it's clear that there's a gendered difference before applying to YC, it seems very difficult to justify an assumption there would be no gendered difference in behavior after applying to YC (or any other investment firm, FirstRound in this case). Given that, the question we were asking becomes much more confusing... a simple bias towards ideas and plans you understand/agree with/are excited by is a gender bias in as much as your gender caused you to like the idea or plan. Removing that bias (supporting plans you understand less, agree with less, or are less excited by) seems like an obviously bad idea.
Returning to the problem: if we accept that this sort of "makes sense to me bias" can be observed when looking for gender biases, we are left in a really hard place. That bias seems to be both a good thing, and confounds the entire analysis. Unless you've controlled for the "makes sense" bias, such analysis will apply pressure for investors to waste money from their perspective. This seems obviously bad.
Core question 2: which biases do we want investors to have?
Investors who knowingly pass up good opportunities on the basis of the founder's gender are punishing themselves worse than any company they pass over -- their competitors who aren't gender biased will get higher returns, and so will have more money to invest in the future. This is to say that gender biases for startup investing are self-correcting. The investors already have their self-interest maximally aligned with not being sexist.
I don't pretend to know which companies are worth investing in more than any other smart technologist. I also don't pretend to know to what extent gender differences cause differences in returns, so my answer is: investors should be as biased (selective about investing) as they see fit. Startups are positive sum for society, and anyone who can find a way to fund more of them profitably is making the world better.
In large part, this is because I find it very unlikely that any modern investor is knowingly sexist -- I think it's much more likely that the sort of "makes sense" bias I discuss above is at play.
Of course, this is an early thought that came from first principals, so counter arguments solicited. Perhaps there is something deeply evil about passing over startups you don't feel comfortable investing in (assuming that comfort has any correlation with founder gender), or perhaps there's some easy fix which makes previously dicy-looking ideas from {other-gender} founders look like obviously good investments. (If you know what that idea is, I'd love to know it too).
5. Thanks to both pg and Chris for the fun math/philosophy problem. :)
Re: The Mathematics of Paul Graham's Bias Test
#75Earlier quoted context omitted.
Dropping outliers is common in statistical analysis.
It's still BS. Outliers are a signal that you don't have a simple, nicely decaying distribution. The right way to deal with outliers is to use a method that acknowledges their existence, not to ignore them. For example, if outliers destroy your OLS linear regression, it's because your error is not normal . That means you need to do Bayesian linear regression with a non-normal error term, not just throw them away.
Depends. Throwing outliers out without thinking is obviously wrong. In many instances outliers can be just invalid measurements and you should ignore them.
Re: The Mathematics of Paul Graham's Bias Test
#76Earlier quoted context omitted.
It's still BS. Outliers are a signal that you don't have a simple, nicely decaying distribution. The right way to deal with outliers is to use a method that acknowledges their existence, not to ignore them. For example, if outliers destroy your OLS linear regression, it's because your error is not normal . That means you need to do Bayesian linear regression with a non-normal error term, not just throw them away.
Depends. Throwing outliers out without thinking is obviously wrong. In many instances outliers can be just invalid measurements and you should ignore them.
A much better approach is to incorporate measurement error into your statistical procedure.
Of course that's usually a lot easier to do with Bayesian techniques...
Re: The Mathematics of Paul Graham's Bias Test
#77Earlier quoted context omitted.
Dropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' entire goal is to find outliers.
Possible. In this case, we're not looking for outliers or measuring based on financial success, but trying to tell if the VC is systematically biased anti-woman. It's not clear that dropping outliers is a bad idea there. It's also not clear it's a good idea, granted.
Well, if you are trying to measure whether men founders or women founders you have funded on average make more, then you would have to include Uber. The real issue with the analysis is that results are unlikely to be statistically significant due to small samples and high variance, which means they are useless.
Re: The Mathematics of Paul Graham's Bias Test
#78Earlier quoted context omitted.
Dropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' entire goal is to find outliers.
Possible. In this case, we're not looking for outliers or measuring based on financial success, but trying to tell if the VC is systematically biased anti-woman. It's not clear that dropping outliers is a bad idea there. It's also not clear it's a good idea, granted.
Well, if you are trying to measure whether men founders or women founders you have funded on average make more, then you would have to include Uber. The real issue with the analysis is that results are unlikely to be statistically significant due to small samples and high variance, which means they are useless.
Re: The Mathematics of Paul Graham's Bias Test
#79Honest question: suppose I pick a plausible sounding h(x), perform the proposed test, and get a vanishingly small p-value. So I feel pretty happy about rejecting the null hypothesis. But the null hypothesis is a conjunction: A group members accepted have cdf a(x), B group members accepted have cdf b(x), and h(x) satisfies various technical conditions related to a( ) and b( ). So when I reject the null, I'm saying the data were unlikely to be generated by the hypothesized process. Couldn't that simply be because I guessed the wrong form of h( )?
Re: The Mathematics of Paul Graham's Bias Test
#80> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…
> If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean) Both the mean and the minimum are a scalar-valued function of the whole sample.
[deleted]