How Not To Sort By Average Rating

The assumption is that there's some constant p underlying probability that a random person will rate a given thing positively. If we observe, for instance, 4 positive and 5 negative reviews or votes, there's a probability distribution (known as a Beta distribution) which tells us what the possible values of p are given the votes we observe: p^4 (1-p)^5. graph: https://www.google.com/search?q=x%5E4+(1-x)%5E5%20from%200%2...

Now if we observe 40 and 50, respectively, the curve looks like this: https://www.google.com/search?q=exp(20+%2B+40+log(x)+%2B+50+...

(I had to do it in the log domain because Google's grapher underflows otherwise -- the 20 is just to make the numbers big enough to graph. The more correct thing involves gamma functions and that just gets in the way right now)

The more you observe, the more sharply peaked the likelihood function is. The funky equation in the article is an approximation to the confidence interval of that graph -- 95% of the probability mass is said to be within those bounds.

It's not a great approximation, for one because the graph is skewed (try it with 10/50) and it assumes that the mean is exactly in the middle of the confidence interval. The correct computation involves the inversion of a messy integral called the incomplete beta function. Scipy has a package which includes betaincinv which solves this more exactly:

>>> import scipy.special

>>> scipy.special.betaincinv(5,6, [0.025, 0.975])

array([ 0.18708603, 0.73762192])

would be the 95% confidence interval for 4 positive and 5 negative votes;

>>> scipy.special.betaincinv(41,51, [0.025, 0.975])

array([ 0.34599562, 0.54754792])

for 40 and 50, respectively.

[edit: apologies, I had to run and get ready for work -- I didn't really have time to make this very comprehensible; but i just now fixed a bug in my confidence interval stuff above]

http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...

Of course, I think the authors miss the point of the algorithm, since I basically wanted a system that is one-sided (i.e. false negatives are OK but false positives are bad).

Also, if you deal with more than two outcomes you might be interested in multinomial confidence intervals, described here:

http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht...

The application to 5-star systems is not straightforward, since it's not clear to me how stars relate to each other. Is it a linear scale? Are they discrete buckets? Or maybe we want to use Tukey's froots and flogs? I'm not sure.

By the way, I'm coming out with a stats app for Mac soon that implements this algorithm and much more. Drop me your email address if interested:

http://wizard.evanmiller.org/

I never said there was. In fact, I praised it as an elegant solution.

It just requires better educators to explain it, with analogies and metaphor.

You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000.

Why would you compromise the data/outcome in a hope to simplify the problem?

Actually, simplying the problem actually reduces the compromise when humans are involved. So, no.

...wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not?

Only if it made economic sense to do so. I am not going to replace 800 workers earning $12/hour with "the best people who understand the domain" because the computer suddenly has formulas that people don't understand. Experience has shown repeatedly that workers, at any level, simply stop caring when they feel powerless by "solutions" like OP's Wilson formula. That abducation of responsibility almost always far outweighs any incremental benefit that a more sophisticated but incomprehensible formula introduces.

Good points, nice discussion, but please, for the sake of this community, don't reply with words like "what?", "Further:", or "Hint:". You made your point without the snarkiness.