Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Mathematics of Paul Graham's Bias Test

chrisstucchio.com

Post reply on HN
Print view View original

11–20 of 84 posts

Re: The Mathematics of Paul Graham's Bias Test

#11

Post by staunch »

lukev wrote: post #10

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performa…

This seems so obvious that it's possible we're missing something. It seems, at best, "A Way to Diff Your Bias", which isn't the same thing as detecting your bias at all. No one has the goal (I hope) of aligning their negative biases with others.

Re: The Mathematics of Paul Graham's Bias Test

#12

Post by danieltillett »

Steko wrote: post #7

> The idea is generally correct - bias in a decision process will be visible in post-decision distributions I find what's wrong with the idea more fundamental, that it talks only about the 'selection process' but in fact bias that impacts success or failure can come at other points.

This is really important. Lets say the whole VC ecosystem is biased against redheads (just to pick a random group). What would happen is the redheads would under perform other groups as they were discriminated against at each stage of the VC lifecycle. They would not show up as a group that over performing later. The only bias you can detect using Paul’s approach is bias that only applies at the initial stage and not later.

Re: The Mathematics of Paul Graham's Bias Test

#13

Post by bsder »

The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place.

"But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason.

If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.

Re: The Mathematics of Paul Graham's Bias Test

#14

Post by nepstein »

lukev wrote: post #10

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performa…

True - though this means only that the signal (outperformance by a group that is the target of discrimination) might not be present (if discrimination continues past initial selection as you point out).

However, the test may still useful to help confirm bias. If outperformance is observed, you can infer one of 3 things is true:

1) there is bias at initial selection but not after (or at least reduced bias)

2) members of the outperforming group are simply stronger performers (different but still interesting)

3) there is no bias at selection but there are affirmative action effects after the initial selection (not obvious why this would be the case)

Re: The Mathematics of Paul Graham's Bias Test

#15

Post by nshepperd »

lukev wrote: post #10

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performa…

The theory presumes a reliable way of measuring "actual performance"; that is, the quantity against which the selection process is supposedly biased. That's a limitation, but I wouldn't say it makes the test "wrong".

It does mean that maybe monetary earnings or anything else sensitive to later-round bias are not the thing to use to measure candidate performance, at least if you're doing this for the social utility.

Of course, if you're only in it to make money, and you're only in charge of the first round... then you really do want just an unbiased evaluation of the (biased) future earnings prospects. So in that case using raw earnings would be correct...

Re: The Mathematics of Paul Graham's Bias Test

#16

Post by escape_goat »

danieltillett wrote: post #12
Steko wrote: post #7

> The idea is generally correct - bias in a decision process will be visible in post-decision distributions I find what's wrong with the idea more fundamental, that it talks only about the 'selection process' but in fact bias that impacts success or failure can come at other points.

This is really important. Lets say the whole VC ecosystem is biased against redheads (just to pick a random group). What would happen is the redheads would under perform other groups as they were discriminated against at each stage of the VC lifecycle. They would not show up as a group that over performing later. The only bias you can detect using Paul’s approach is bias that only applies at the initial stage and not…

Just to cross the beams of pedantry here for a moment, a widespread and well known --- if less than serious or systemic --- cultural/social bias against red haired people, probably first coming into public consciousness in North America due to the infamous South Park 'Ginger' episode, has in fact primed you to select "redheads" as a non-contentious example of a plausibly ethnic group that might be discriminated against, something that every red-haired person knows, although you apparently do not. This means that the choice is statistically insensitive and the social methodology is poor.

Actually, choosing an identifiable group at random would be both socially and statistically unwise, as, following Patero distribution, there are vastly more minority/extreme minority distinguishable groups of people than there are majority/significant minority ones; this means, firstly, that any group randomly selected with equal biasing between all groups has a high probability of being subject to actual discrimination, mooting any social benefit of choosing a group at random; secondly, that the generalizable qualities of the group chosen would therefore have a distribution with very little deviation (if I'm using my terms correctly) and would be highly predictable, thereby obviating any possible statistical benefit of doing so.

Re: The Mathematics of Paul Graham's Bias Test

#17

Post by [deleted] »

nshepperd wrote: post #8
LordHumungous wrote: post #6

Lots of math in here premised on shaky foundations: >Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90% >The mean of group B is not lower because of bias (which would be reflected near x=80), but because the very best members of group B are simply not as good as the very best members…

Read it again. He's talking about the counterexample there. It's a hypothetical.

[deleted]

Re: The Mathematics of Paul Graham's Bias Test

#18

Post by [deleted] »

nshepperd wrote: post #8
LordHumungous wrote: post #6

Lots of math in here premised on shaky foundations: >Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90% >The mean of group B is not lower because of bias (which would be reflected near x=80), but because the very best members of group B are simply not as good as the very best members…

Read it again. He's talking about the counterexample there. It's a hypothetical.

[deleted]

Re: The Mathematics of Paul Graham's Bias Test

#19

Post by LordHumungous »

nshepperd wrote: post #8
LordHumungous wrote: post #6

Lots of math in here premised on shaky foundations: >Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90% >The mean of group B is not lower because of bias (which would be reflected near x=80), but because the very best members of group B are simply not as good as the very best members…

Read it again. He's talking about the counterexample there. It's a hypothetical.

Yes I get that. The crux of his argument is:

>Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C

His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math.

Re: The Mathematics of Paul Graham's Bias Test

#20

Post by Steko »

danieltillett wrote: post #12
Steko wrote: post #7

> The idea is generally correct - bias in a decision process will be visible in post-decision distributions I find what's wrong with the idea more fundamental, that it talks only about the 'selection process' but in fact bias that impacts success or failure can come at other points.

This is really important. Lets say the whole VC ecosystem is biased against redheads (just to pick a random group). What would happen is the redheads would under perform other groups as they were discriminated against at each stage of the VC lifecycle. They would not show up as a group that over performing later. The only bias you can detect using Paul’s approach is bias that only applies at the initial stage and not…

To take your example one step further, the redhead performance would be held up under pg's rubric as evidence that non-redheads are biased against and so redheads may fall into a vicious circle of deepening discrimination.
Post reply on HN