11–20 of 224 posts
A related observation (which I've been making for a long time) is that the absence of mediocre women in positions of power is strong evidence of bias. Men can succeed when they're mediocre, but women have to be exceptional. Likewise for minorities.
I think the first footnote in this is extremely valid. It all depends on what performance metric the selection process is identifying compared to the performance metric you use to determine success. I would suspect the larger issue is that people are probably much worse at identifying what performance metrics for selection convert to their respective performance for success.
If students of Asian origin outperform the whole student body, can we conclude admissions folks are biased against students of Asian origin?
The first is if you have multiple people accepting applicants and some of them are biased to the point of not accepting applicants of particular types. That means all the applicants that are discriminated against that did make it were simply selected by people who weren't biased, and therefore won't outperform anyone.
The second is if the actual selection process is somewhat random instead of being based on pure performance. The ones who make it through that process won't necessarily perform any better, they'll just be luckier.
The third is if the application process accepts everyone equally, and then randomly prunes out people according to a bias. This is similar to the second except the acceptance criteria is still performance-based, but because it randomly throws out people (instead of throwing out low-performers), the remaining people are still going to perform the same as those who were not pruned.
The first footnote on the page also points out that if the selection criteria are different for the different groups then this process won't work, which seems like a pretty important caveat that I wish was in the article proper. One really common form of bias (especially in tech) is being biased against women, and that's also a situation where it's very common to (unconsciously or otherwise) use appearance in judging female applicants but ignore appearance for male applicants.
Except if you want to use statistics to measure bias, you need a statistically significant sample. And actually, if you are studying complex human affairs, with a hundred different variables, you need more than statistical significance, you need a sensitivity analysis. It is similar to nutrition studies. There are so many variables at play that something can always be found to increase or decrease your risk of cancer by 50%. You really only need to pay attention when statistics show an order-of-magnitude correlation, as with the link between smoking and lung cancer.
With the First Round Capital data, they excluded Uber from their calculations, because it would skew everything. If a single data point can switch your findings to be opposite, then you just have to admit that you do not have enough data to make determination one way or another. In science it is sometimes ok to exclude an outlier, since it often indicates a measurement error. But in venture capital, you make most of your money off of the Uber-like outliers. So if you are trying to study the data to be the best venture capitalist possible, throwing out outliers is not valid.
Also, the initial premise is incorrect too. You cannot measure bias by comparing average results, because the average is not the marginal. Consider PG's footnote: "Although I used female founders as an example because that is a kind of bias people often talk about, the most striking thing was the degree to which First Round undervalue founders who went to elite colleges." Does he honestly believe that First Round is biased against founders from elite colleges?
At my last company my sense was that the MIT grads were better than the average programmer. So were we biased against MIT grads? Should we have hired more MIT grads until the average performance of MIT grads overall equaled the average performance of an employee overall? Should we have done more outreach to MIT? Should the industry as a whole hired more MIT grads?
If a talent distribution has a bunch of elite, and then a steep drop-off filled with "pretenders", then you can get this type of effect without being biased.
When we got an elite MIT grad, we hired them. When we got a "pretender", someone who was trading on the name but did not put in the work, we rejected them. And yes, I personally saw MIT grads that did terrible on simple coding exercises.
So even though the average MIT grad we hired was better than the average programmer at our company, there was no way to alter our hiring process to get more MIT grads. If we hired the marginal MIT grad that we rejected, we would have been worse off. Now we could do more outreach to MIT, and we did, but that is a highly competitive process. There were diminishing marginal returns to how much outreach we can do to get more applicants.
The statistical illiteracy of PG's post is simply stunning. Imagine a YC company gets a 100% ROI from PPC ads, and a 50% ROI from banner ads. Are they biased against PPC ads? Should they buy more PPC ads? Such an analysis is ridiculous. You look at what you are spending on the marginal PPC ad, and you stop spending when the ROI on the marginal ad is at zero, regardless of what the average is. That one advertising channel has a higher ROI on average does not mean that the company is biased against that channel.
There's a few trivial ways to be biased that would not be detected in this way. The first is if you have multiple people accepting applicants and some of them are biased to the point of not accepting applicants of particular types. That means all the applicants that are discriminated against that did make it were simply selected by people who weren't biased, and therefore won't outperform anyone. The second is if the…
There's a few trivial ways to be biased that would not be detected in this way. The first is if you have multiple people accepting applicants and some of them are biased to the point of not accepting applicants of particular types. That means all the applicants that are discriminated against that did make it were simply selected by people who weren't biased, and therefore won't outperform anyone. The second is if the…
If noise is present, then you get convolve(H(x-C)f(x), k(x)) instead, where k(x) is the pdf of the noise distribution.
You'll need more samples to measure this, but it's completely measurable via Graham's method.
There's a few trivial ways to be biased that would not be detected in this way. The first is if you have multiple people accepting applicants and some of them are biased to the point of not accepting applicants of particular types. That means all the applicants that are discriminated against that did make it were simply selected by people who weren't biased, and therefore won't outperform anyone. The second is if the…
I think the first footnote in this is extremely valid. It all depends on what performance metric the selection process is identifying compared to the performance metric you use to determine success. I would suspect the larger issue is that people are probably much worse at identifying what performance metrics for selection convert to their respective performance for success.
If students of Asian origin outperform the whole student body, can we conclude admissions folks are biased against students of Asian origin?
The "high performance means there's a bias" theory only works if you assume that everyone is starting from the same social, cultural, and physical baseline. They aren't.
Maybe a better metric would be if there were no mediocre data points among a certain group; that would be more evidence (but still not necessarily good evidence) that you have to exceptional to get attention and overcome the "bias barrier", not simply that most of a certain type of performer does better than a different type.
For example, assume that we find that founders who won a MacArthur "genius" grant outperformed the others. Further assume that there are only a limited number of such founders, and that all available were selected. Certainly one wouldn't want to conclude in this case that there is a bias against MacArthur fellows.
That seems obvious, but it gets trickier once you have lots of factors involved. What if the group you find to outperform consists of female founders with a PhD, substantial industry experience, and red hair[1]. Can you conclude that the process is biased against females? Males with PhD's? Anyone with red hair? Generally no, unless you are willing to assume that all of your factors are causal.
Worse, you can't even assume that it's biased against people with all of the measured factors unless you also assume that all unmeasured factors are randomly distributed. If it turns out that "a positive mental attitude" is an unmeasured but defining characteristic of success, if the interviewers rejected applicants who had less of this but you failed to include this in your category, you would be wrong to conclude that there is an unfair bias.
[1] http://www.nature.com/nature/journal/v453/n7194/full/453562a...
Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80% or better chance of graduation. Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%, with the same average but less variation in group B.
Now suppose that we select without bias or inaccuracy all the applicants that have an 80% or better chance of graduation. That means we select a subset of A with a range of 80% to 100% and a subset of B with a range from 80% to 90%. The average graduation rate of scholarship winners from group A will be 90% and that from group B will be 85%.
But we haven't been biased against A. We've selected according to the exact same perfect evaluation process and criterion from both groups. It was just their prior distribution that was different.
The actual applicant groups for jobs or financing in the real world, when they are divided by demographic factors like age, sex, race, and educational level, will almost always manifest different variances in success levels even when the averages are the same. That makes this test useless and mathematically illiterate.
And when we use a normal distribution, as we should always expect given the central limit theorem, the mathematical problems get even more intense.
This short comment is not up to pg's usual high standards for his essays.