The Mathematics of Paul Graham's Bias Test • Hacker News Board

Re: The Mathematics of Paul Graham's Bias Test

#81

Post by dlss » Fri, Nov 06, 2015, 4:06 PM UTC

Scea91 wrote: post #75

yummyfajitas wrote: post #56

Earlier quoted context omitted.

It's still BS. Outliers are a signal that you don't have a simple, nicely decaying distribution. The right way to deal with outliers is to use a method that acknowledges their existence, not to ignore them. For example, if outliers destroy your OLS linear regression, it's because your error is not normal . That means you need to do Bayesian linear regression with a non-normal error term, not just throw them away.

Depends. Throwing outliers out without thinking is obviously wrong. In many instances outliers can be just invalid measurements and you should ignore them.

> In many instances outliers can be just invalid measurements and you should ignore them.

signal[i] = value[i] + noise[i].

If you know that value[i] == NaN, then by all means throw out signal[i]. If value[i] != NaN, then you're better off modeling error[i], and using that model to give you information about value[i] as yummyfajitas suggests.

This is trivial to see if noise[i] == 0, but for some reason becomes progressively harder for people as noise[i] increases.

Re: The Mathematics of Paul Graham's Bias Test

#82

Post by rcthompson » Fri, Nov 06, 2015, 4:51 PM UTC

nazka wrote: post #48

rcthompson wrote: post #47

Earlier quoted context omitted.

The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was u…

It's very interesting. And what are the most suitable formulas we can use to measure how robust it is?

The robustness of a sample min or max for a continuous distribution is basically proportional to the density of the distribution at that extremum. A steep or vertical drop-off at the edge of the distribution is the ideal case.

The article gives a formula for the statistical power of the hypothesis test derived from the sample min. It depends on the function h(x), whose purpose is to establish a lower bound on the density of the distribution at the min, and hence a lower bound on the robustness of the sample min as an estimator.

Re: The Mathematics of Paul Graham's Bias Test

#83

Post by wattle_park » Sat, Nov 07, 2015, 5:06 AM UTC

To use maths/statistics to reason that YC is not biased against certain groups of applicants is amusing. But to even consider that technical female founders are weak candidates is disappointing.

The sample population was chosen by specific type of groups of partners. There is no female technical partners in the group. As a female technical founder, I am not interested in building 'tea-making bot', sandwich making bot or selling organic condom. IMHO we have different views in looking at problems and solving them. Without having female technical founder as partner, YC would be perceived to be biased.

The algorithm of selecting promising candidates will vary once there is a variety of partners.

Re: The Mathematics of Paul Graham's Bias Test

#84

Post by Pyxl101 » Sat, Nov 07, 2015, 10:30 PM UTC

venomsnake wrote: post #43

Pyxl101 wrote: post #41

Earlier quoted context omitted.

Why is it appropriate to drop outliers? (The fact that something is common does not make it a good thing.)

Statistics 101. When you have samples you throw away the highest and lowest member, to counteract some random occurrence. The mean net worth of the patrons in any restaurant carlos slim frequents rises substantially when he is there.

Yes, because that is how mean net worth is defined. I don't see specifically what that argues against, except that mean is not the best indicator to use in all situations; perhaps a different indicator is appropriate, such as the income per patron by percentile. 100th percentile will be Carlos Slim, but 99th percentile and lower will be other patrons.

If Carlos Slim actually does frequent the casino, then his attendance is an important part of understanding the situation.