How Not To Sort By Average Rating • Hacker News Board

Re: How Not To Sort By Average Rating

#21

Post by brown9-2 » Tue, Apr 03, 2012, 2:22 PM UTC

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

And as soon as humans saw that decision making formulas were too complex to understand, they relieved themselves of responsibility for those decisions. "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

And somehow the solution isn't to have the computer output a detailed description explaining why it arrived at the decision that it did?

Re: How Not To Sort By Average Rating

#22

Post by NathanRice » Tue, Apr 03, 2012, 2:22 PM UTC

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer.

Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item. The benefit of a posterior here is that it lets you rank items by thinking in terms of the probability that the viewer would rank one item higher than another at random. By adjusting the magnitude of the alpha parameter to the dirichlet prior, you adjust your sensitivity to small numbers of observations. A small initial alpha will lead to rapid changes in the posterior upon observing ratings, whereas a large alpha requires a significant body of evidence.

The best part of the multinomial model with conjugate dirichlet prior is that the math is REALLY simple. The normalizing constant for the dirichlet distribution looks scary when stated in terms of the gamma function, but given this is the discrete case, just pretend everywhere you see the gamma(x), it is replaced with (x - 1)! and you will be ok.

Let me know if you would like to learn more, I would be happy to help.

Re: How Not To Sort By Average Rating

#23

Post by brown9-2 » Tue, Apr 03, 2012, 2:24 PM UTC

yariang wrote: post #5

While it is good to look at these sorts of mathematically rigorous algorithms, I think I would be frustrated if it was used everywhere. Or, well, maybe not me perhaps, but a non technical user. The beauty of the second algorithm for rating products is that it is straightforward. Having never seen it before I can deduce that 5 stars come before 4 stars and more reviews come before fewer. If I want to skip ahead to the…

But how is Simple Stupid in the Amazon case a better output for the user? Do you, as an Amazon shopper, really believe that the item with one 5-star review is a better bet for you than the item with 580 reviews and an average of 4.5-stars?

Re: How Not To Sort By Average Rating

#24

Post by joshuahedlund » Tue, Apr 03, 2012, 2:25 PM UTC

For those who do not understand the Wilson algorithm, see this post which was on HN recently, explaining how it works in a little more detail: http://amix.dk/blog/post/19588

(I agree with other commenters that it is complicated and lacks common sense to average users, but I feel like I have a general understanding of the concept thanks to the above link)

Re: How Not To Sort By Average Rating

#25

Post by brown9-2 » Tue, Apr 03, 2012, 2:26 PM UTC

jwr wrote: post #14

I implemented this in a rating system once. Got multiple bug reports, people complained that the system calculates averages wrong, because there are two ratings and the average is obviously not the number they are seeing.

I'm curious, who filed the bugs - the users of the system or the fellow builders of the system? If you rolled back this algorithm based on the bug reports, it sounds like you might have ended up listening to the prior bias of the people on your team, rather than the feedback of the people meant to use the system.

Re: How Not To Sort By Average Rating

#26

Post by jwr » Tue, Apr 03, 2012, 2:29 PM UTC

edash wrote: post #15

jwr wrote: post #14

I implemented this in a rating system once. Got multiple bug reports, people complained that the system calculates averages wrong, because there are two ratings and the average is obviously not the number they are seeing.

Why show the second figure to the user? They don't need to see the calculation you made to determine the sort. Just sort the objects in the order determined by this formula and only show the ratings given by users in the interface.

This was a system where I was supposed to display a star-rating (1-5 stars, with fractional stars as well) for each item.

The people reporting the problems were the users, and yes, this was bias, as they expected averages. That was exactly my point — while the statistics behind this method are sound, it is not what people expect. Building systems that don't do what people expect is difficult.

Re: How Not To Sort By Average Rating

#27

Post by raldi » Tue, Apr 03, 2012, 2:29 PM UTC

imajes wrote: post #19

raldi wrote: post #13

Earlier quoted context omitted.

Instead of displaying stars, Amazon could display a percentage, which under the hood represents the Wilson confidence number. It would be totally intuitive to browse: first come all the 100% items, then the 99's, and so on.

er... and the problem with bucket categorizing? 80-100% = * * * * * 60%-79% = * * * * etc..

How is it a problem if the five-star reviews display first, then the four-star, and so on?

Re: How Not To Sort By Average Rating

#28

Post by PakG1 » Tue, Apr 03, 2012, 2:29 PM UTC

This is awesome. This is perfect for what we need for our startup. We are going to use this. We won't need to worry about the negative aspects listed in these comments due to our use case. Wow. Thanks, HN. :)

Re: How Not To Sort By Average Rating

#29

Post by mistercow » Tue, Apr 03, 2012, 2:32 PM UTC

NathanRice wrote: post #22

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer. Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item.…

> this is one of those cases where a Bayesian treatment is conceptually much clearer

Is there any other sort of case?

Re: How Not To Sort By Average Rating

#30

Post by raldi » Tue, Apr 03, 2012, 2:36 PM UTC

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.