Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Mathematics of Paul Graham's Bias Test

chrisstucchio.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: The Mathematics of Paul Graham's Bias Test

#41

Post by Pyxl101 »

ceph_ wrote: post #24
bsder wrote: post #13

The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place. "But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason. If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.

Dropping outliers is common in statistical analysis.

Why is it appropriate to drop outliers? (The fact that something is common does not make it a good thing.)

Re: The Mathematics of Paul Graham's Bias Test

#42

Post by JDDunn9 »

haberman wrote: post #28

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…

You could use the mean of the bottom decile.

Re: The Mathematics of Paul Graham's Bias Test

#43

Post by venomsnake »

Pyxl101 wrote: post #41
ceph_ wrote: post #24

Earlier quoted context omitted.

Dropping outliers is common in statistical analysis.

Why is it appropriate to drop outliers? (The fact that something is common does not make it a good thing.)

Statistics 101. When you have samples you throw away the highest and lowest member, to counteract some random occurrence. The mean net worth of the patrons in any restaurant carlos slim frequents rises substantially when he is there.

Re: The Mathematics of Paul Graham's Bias Test

#44

Post by igonvalue »

It's since been dressed up with some mathiness, but this idea was originally proposed in the comment threads of pg's original article. [0] See the responses there for a few reasons why it just won't work.

To be concrete, assuming "performance" is measured as return on investment, min(performance) will always go to to -100% (i.e., bankruptcy) with a large enough sample size.

[0] https://news.ycombinator.com/item?id=10484200

Re: The Mathematics of Paul Graham's Bias Test

#45

Post by lpage »

Maybe I'm missing something but this seems like a pretty perfect application for the bootstrap - a remarkably intuitive but powerful framework. Without loss of generality, imagine that you have two populations, A and B, and that you want to test some hypothesis about a statistic of A being different from a statistic of B (mean, in this case). Using the simplest form of the bootstrap you would do the following:

1. Pool and randomly label the data from A and B 2. Sample with replacement and form two partitions of the same cardinality as the original A and B groups 3. Compute the differences in mean 4. Rinse and repeat millions of times to form a distribution of mean differences 5. Check if the observed difference in means (from the true A/B labels) is statistically significant relative to the distribution found in (4)

This has some problems with fat tailed distributions but tends to work great otherwise. It's so simple that it avoids a host of pitfalls that can arise with other resampling schemes (what's being proposed is a type of resampling), and I love that it makes basically zero assumptions on the underlying data.

Re: The Mathematics of Paul Graham's Bias Test

#46

Post by hinkley »

lukev wrote: post #10

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performa…

It's almost as if they're trying to apply a technical solution to a social problem...

Re: The Mathematics of Paul Graham's Bias Test

#47

Post by rcthompson »

haberman wrote: post #28

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…

The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was used by the Allies in World War 2 to estimate the total number of German tanks by sampling the serial numbers from destroyed tanks[2].)

Of course, a real thresholding process would not be perfect, so the lower bound of the distribution of accepted candidates would not be a perfect vertical cutoff as in the examples. Just like any process that adds additional variation to the data, this would reduce the statistical power. You could accept more bias in return for lower variance in your test by taking, say, the 5th percentile instead of the sample minimum as your test statistic. (You can think of the sample minimum as the zeroth percentile.)

[1] https://en.wikipedia.org/wiki/Uniform_distribution_%28contin...

[2] https://en.wikipedia.org/wiki/German_tank_problem

Re: The Mathematics of Paul Graham's Bias Test

#48

Post by nazka »

rcthompson wrote: post #47
haberman wrote: post #28

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…

The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was u…

It's very interesting. And what are the most suitable formulas we can use to measure how robust it is?

Re: The Mathematics of Paul Graham's Bias Test

#49

Post by notahacker »

JDDunn9 wrote: post #42
haberman wrote: post #28

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…

You could use the mean of the bottom decile.

That would probably be a sensible stat for measuring marginal applications to a school. But when it comes to measuring portfolio startup companies, none of the bottom decile are expected to be worth anything in the medium run, and all the results that actually matter are in the top quartile...

Re: The Mathematics of Paul Graham's Bias Test

#50

Post by nicolapede »

haberman wrote: post #28

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or…

> If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean)

Both the mean and the minimum are a scalar-valued function of the whole sample.

Post reply on HN