Urban Dictionary no longer sorts by positive - negative, see e.g. http://www.urbandictionary.com/define.php?term=usa. I don't know what they use now.
How Not To Sort By Average Rating
51–60 of 159 posts
Re: How Not To Sort By Average Rating
#52There are a lot of comments complaining about how complicated the math is. This shouldn't be all that hard to understand. The assumption is that there's some constant p underlying probability that a random person will rate a given thing positively. If we observe, for instance, 4 positive and 5 negative reviews or votes, there's a probability distribution (known as a Beta distribution) which tells us what the possible…
Very interesting. So would you say developers should probably use the incomplete beta function, rather than Ev's method? Or is it too computationally expensive?
I haven't investigated it in depth -- quadrature over a single variable as this is can be pretty quick to compute. Not sure how scipy does it.
Anyway, I personally think 95% confidence intervals are a crutch. The correct Bayesian approach is to consider two items, each with their own up and down votes, and integrate over all possible values for p1 and p2 (being the underlying probabilities of upvotes for item 1 and 2, respectively) over the observed data, and compute the likelihood of superiority of p1 over p2.
How to turn that into an actual ranking function? No idea. I doubt it would work, but you could compute against a benchmark distribution (i.e. the uniform 0-1 distribution).
If you do that, it probably turns out that your ranking function is the mean of the Beta distribution, which is simple: (U+1)/(U+D+2) where U and D are the upvote/downvote counts [note: we started with the prior assumption that p could be anywhere between 0 and 1, uniformly]. Basically, the counts shrink towards 1/2 by 1. This is a hell of a lot less complicated, and it achieves the goal of ranking different items by votes pretty well with more votes being better.
Re: How Not To Sort By Average Rating
#53I implemented this algorithm by using likes/views in stead of positive/negative votes on http://mediaqueri.es/popular/ and have been quite happy with the results.
Re: How Not To Sort By Average Rating
#54While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer. Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item.…
You could also go one step further on the Bayesian path and infer even alpha from the data on your site, and introduce a loss function on your ordering. Or you could do a semi-frequentist thing and simplify your math by using MAP estimates to rank. Basically instead of score = #pos/(#pos + #neg), it becomes score = (#pos+x)/(#pos+x + #neg+y), where you choose x and y to suit your needs. You could choose x/y in propor…
Absolutely! I just wrote a reply where I alluded to that, unfortunately I didn't refresh and see this post or I would have just plugged you instead.
The benefit of the Bayesian treatment here that I want to drill down on is how natural it is to adjust the prior to capture your beliefs about how items should be perceived in the presence of incomplete information. The frequentist approach is fine, but it does not provide such a pleasant, intuitive knob to tune.
Re: How Not To Sort By Average Rating
#55Earlier quoted context omitted.
Here's a paper proposing a solution in that space, and which also compares itself to the article linked here (kind of nice to see... papers sometimes fail to cite stuff that's "only" posted online rather than properly published, even if the authors know about it and it's quite relevant): http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... I emailed Miller a while ago to see what he thought of this reply,…
I don't think you have to resort to any overly complex machinery to achieve similar behavior. The simplest approach is to just use a non uniform prior. His pessimistic bound could be emulated by having an initial alpha that places more weight on low star ratings. The intuitive interpretation of that being "things are probably bad unless proven good" roughly. Another option would be to generate the prior based on the…
I agree the lower bound of a test statistic is a pretty indirect way of encoding intuitions. Somehow I tend to find loss functions the conceptually clearest way of encoding preferences about inference outcomes, though. But, in this case my feeling in that direction isn't very strong, and the priors-based solution seems fine.
Re: How Not To Sort By Average Rating
#56I've always thought about this, and to me, a very simple (though slightly inaccurate) solution would be to sort using this formula:
(TotalScore - 1) / MaxPossibleScore
Such that (using the Amazon examples from the article):
((2 * 5) - 1) / 10 = 9/10 = 90%
((100 * 5) + (1 * 1) - 1) / 505 = 500/505 = 99%
Re: How Not To Sort By Average Rating
#57I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…
Math is hard, let's go shopping!
If you think this is too difficult then maybe you shouldn't try to design scoring/voting systems.
Re: How Not To Sort By Average Rating
#58My favorite part was looking at the code and seeing a variable named `phat', and then looking up at the equation to find `p^' (p-hat).
Re: How Not To Sort By Average Rating
#59Date: 9:12 AM Wednesday, April 4, 2012
From: the boss
To: dev3
Subject: URGENT - front page showcase selection broken!!
Body: Hey bro, I was looking into it, and our ratings average equation is totally busted and products with just a few ratings are hogging space from proven winners when it's just a sample bias. This is costing us money and needs to be fixed NOW.
I'd like this up before our morning meeting so I can boast about it and you'll get credit too, as this should massively increase our conversions right away by putting BETTER products right on the front page.
this should get you started: http://evanmiller.org/rating-equation.png
I'm sure you'll figure it out. If you could do an A/B test for bragging rights too that would MASSIVELY rock. Thanks!!!
Rock on,
Boss
Re: How Not To Sort By Average Rating
#60Earlier quoted context omitted.
You could also go one step further on the Bayesian path and infer even alpha from the data on your site, and introduce a loss function on your ordering. Or you could do a semi-frequentist thing and simplify your math by using MAP estimates to rank. Basically instead of score = #pos/(#pos + #neg), it becomes score = (#pos+x)/(#pos+x + #neg+y), where you choose x and y to suit your needs. You could choose x/y in propor…
Absolutely! I just wrote a reply where I alluded to that, unfortunately I didn't refresh and see this post or I would have just plugged you instead. The benefit of the Bayesian treatment here that I want to drill down on is how natural it is to adjust the prior to capture your beliefs about how items should be perceived in the presence of incomplete information. The frequentist approach is fine, but it does not provi…
Well, it depends on how you look at it. The Bayesian MAP approach is basically the same as the frequentist approach with made up data. Instead of using the maximum likelihood estimator p = pos/(pos+neg) you pretend that each new post already has some up/down votes by default, and then you use simply p = pos/(pos+neg). Seems like an even more intuitive explanation of the Bayesian knob to me! Rather than an abstract "alpha" to a "Dirichlet prior", you get something concrete (the number of made up votes). And you get a simple formula, which some would find desirable.
But I agree that the Bayesian approach is conceptually much cleaner. IMO the frequentist approach is just computational corner cutting for when the math in the Bayesian approach gets too involved, which is sometimes useful. What's nice about the Bayesian approach is that you state your assumptions and then it's just turning the math machinery. In contrast, in the frequentist approach the assumptions are interwoven and hidden in arbitrary choices in how the math is done (And then they claim that Bayesians are subjective! It's just that Bayesians admit that they are subjective. Frequentists try to hide the fact that they are more subjective in the math). The not so nice thing is that turning the math machinery is not always so easy and does not always produce fast algorithms. That's where maximum likelihood and friends come in, but I'd view them as an approximation to Bayesian methods.