Live data from Hacker News

How Not To Sort By Average Rating

evanmiller.org

81–90 of 159 posts

Re: How Not To Sort By Average Rating

#81

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer. Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item.…

I would absolutely love to learn more. I've been trying to solve some novel NLP and machine learning problems lately but my lack of statistical knowledge is becoming apparent the further along I get.

Do you have any recommendations for a good introductory treatment of Bayesian statistics?

Re: How Not To Sort By Average Rating

#82

Original author here. For the academically inclined, there is a critique of this approach in this paper: http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... Of course, I think the authors miss the point of the algorithm, since I basically wanted a system that is one-sided (i.e. false negatives are OK but false positives are bad). Also, if you deal with more than two outcomes you might be interested in mul…

I really appreciated the post. Unfortunately we're using 5 stars, and need to do it a bit less wrong.

The main thorniness of 5 stars is that you have to answer the question of what the difference in star ratings actually mean. Is going from one star to two stars the same as four to five? Probably not, based on the way users rate, which means an algorithm like the arithmetic mean that treats them the same is wrong.

Personally, I think it's very reasonable to treat rating stars the way we should treat grades: as ordinal data, where we know that a higher rating is better but assume nothing beyond that. The difference between an F and a D is not the same as an A and a B, and the same is likely true of 1-2 stars and 4-5 stars.

I have made an attempt at applying this idea to the Ubuntu App Store's rating algorithm. I'm very much interested in comments. https://bugs.launchpad.net/ubuntu/+source/software-center/+b...

Re: How Not To Sort By Average Rating

#83
Here's an even simpler way to think about it: it's the left point of the standard 95% confidence interval from the Central Limit Theorem plus a hack for small sample sizes. The Wikipedia page says the hack is almost equivalent to estimating p = (X+2)/(n+4) i.e. assuming each item starts with two upvotes and two downvotes.

Re: How Not To Sort By Average Rating

#84

Earlier quoted context omitted.

Here's a paper proposing a solution in that space, and which also compares itself to the article linked here (kind of nice to see... papers sometimes fail to cite stuff that's "only" posted online rather than properly published, even if the authors know about it and it's quite relevant): http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... I emailed Miller a while ago to see what he thought of this reply,…

I don't think you have to resort to any overly complex machinery to achieve similar behavior. The simplest approach is to just use a non uniform prior. His pessimistic bound could be emulated by having an initial alpha that places more weight on low star ratings. The intuitive interpretation of that being "things are probably bad unless proven good" roughly. Another option would be to generate the prior based on the…

There's a distinct difference in the asymptotic behavior though between the lower bound and the prior. The lower bound goes to the mean as 1/sqrt(n), the prior goes to the mean as 1/n.

That makes for a pretty significant difference in practice, and I'm not sure which is preferable.

Re: How Not To Sort By Average Rating

#85
post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

Meh. "Why did you return the list of reviews in this order?" - "Here's the link to Wikipedia explaining it".

The formulas aren't "too complex to understand", the people are being too lazy to take the time and understand them. No reason to dumb it down and ruin it for the rest of us.

Re: How Not To Sort By Average Rating

#86
post #30
post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

I think you're overlooking a critical component of the Google equation: the supply.

Webmasters * constantly* complain about the impenetrability of the 'algorithm' and how it constantly changes for secret reasons.

So you're right in a way---the demand side doesn't care how you manage to curate supply, but supply is intimately aware of your curation and worries about being unfairly demerited.

Re: How Not To Sort By Average Rating

#87
post #13
post #5

While it is good to look at these sorts of mathematically rigorous algorithms, I think I would be frustrated if it was used everywhere. Or, well, maybe not me perhaps, but a non technical user. The beauty of the second algorithm for rating products is that it is straightforward. Having never seen it before I can deduce that 5 stars come before 4 stars and more reviews come before fewer. If I want to skip ahead to the…

Instead of displaying stars, Amazon could display a percentage, which under the hood represents the Wilson confidence number. It would be totally intuitive to browse: first come all the 100% items, then the 99's, and so on.

You can't use Wilson's confidence with a star-rating system. Wilson's only works for binary systems.

Instead you could use a weighted baysian rating:

br = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes)

Re: How Not To Sort By Average Rating

#88
post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

>What human can understand it?

Lets start with Wilson's midpoint, since that's just high school math.

     def mid(upvotes:Int, downvotes:Int) = {
      val total = upvotes+downvotes+0.0
    
      val up = upvotes/total
      val half = 0.5
    
      val a = total/(4+total)
      val b = 4/(4+total)
    
      a * up + b * half
     }
So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes.

It should be very clear that if the total becomes large, a goes to 1 and b to zero. At that point you end up using the proportion of upvotes, just like Amazon.

Now, lets bring in the entire confidence interval.

     def wilson(upvotes:Int, downvotes:Int) = {
      val z = 1.96
      val n = upvotes+downvotes+0.0d
      val phat = upvotes/n

      val lower = (phat + z*z/(2*n) - z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
      val upper = (phat + z*z/(2*n) + z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
      (lower,upper)
      }
Using the author's data points, the results look like this:

     val itemVotes = List((600,400),(5500,4500),(2,0), (100,1))
     itemVotes.foreach( x => {
      val s = mid(x._1,x._2)
      val w = wilson(x._1,x._2)
      printf("Up:%5d\tDown:%5d\tMid:%.3f\tW:[%.3f,%.3f]\n", 
      x._1, x._2,s,w._1,w._2)
      })

       scala> Up:  600	Down:  400	Mid:0.600	W:[0.569,0.630]
        Up: 5500	Down: 4500	Mid:0.550	W:[0.540,0.560]
        Up:    2	Down:    0	Mid:0.667	W:[0.342,1.000]
        Up:  100	Down:    1	Mid:0.971	W:[0.946,0.998]
Basically, we prefer the lower bound of the confidence interval instead of the midpoint of that same interval.

Re: How Not To Sort By Average Rating

#90

Earlier quoted context omitted.

I don't think you have to resort to any overly complex machinery to achieve similar behavior. The simplest approach is to just use a non uniform prior. His pessimistic bound could be emulated by having an initial alpha that places more weight on low star ratings. The intuitive interpretation of that being "things are probably bad unless proven good" roughly. Another option would be to generate the prior based on the…

There's a distinct difference in the asymptotic behavior though between the lower bound and the prior. The lower bound goes to the mean as 1/sqrt(n), the prior goes to the mean as 1/n. That makes for a pretty significant difference in practice, and I'm not sure which is preferable.

You are absolutely correct that they are not mathematically identical. I struggled to word it in a way that would not mislead people, the distinction is important to emphasize.
Post reply on HN