Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not To Sort By Average Rating

evanmiller.org

Post reply on HN
Print view View original

111–120 of 159 posts

Re: How Not To Sort By Average Rating

#111

Post by gus_massa »

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

To get a general idea, you can compare this formula with the simple version, that is * mu - 3 sigma* (be careful when the number of samples is small), where mu is the average and sigma is the standard variation, and 3 is a parameter that depends on the confidence (~93%). When the number of samples (n) is big, then the two formulas give similar results.

Re: How Not To Sort By Average Rating

#112

Post by moultano »

NathanRice wrote: post #90
moultano wrote: post #84

Earlier quoted context omitted.

There's a distinct difference in the asymptotic behavior though between the lower bound and the prior. The lower bound goes to the mean as 1/sqrt(n), the prior goes to the mean as 1/n. That makes for a pretty significant difference in practice, and I'm not sure which is preferable.

You are absolutely correct that they are not mathematically identical. I struggled to word it in a way that would not mislead people, the distinction is important to emphasize.

It has a really big effect I think on the tone of what gets selected at the top. The lower bound prefers things that are preferred by a majority and very popular. The prior method prefers things that are completely un-objectionable and liked by just enough people to be sure of that. My hunch is that with the lower bound you get more interesting things bubbling to the top because it puts a stronger emphasis on popularity.

In all of these models, the giant variable that is completely ignored is the actual choice to rate something at all, versus skipping over it and reading the next one. That's a very significant decision that the user makes. The behavior of each of these systems w.r.t that effect will be the dominant thing differentiating them.

Re: How Not To Sort By Average Rating

#113

Post by dxbydt »

davidw wrote: post #110
dpritchett wrote: post #75

Earlier quoted context omitted.

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And…

That much was fairly clear on the site. I'm talking about the math itself.

With zero knowledge of statistics, here's how to think about it - Wilson wants to construct some score. That score can be as low as 0 and as high as 1. So he wants some interval [x,y]. The center of that interval would obviously be c = (x+y)/2. Wilson first decides what c to pick. So Wilson says, the center c must be decided by the proportion of upvotes. But then he thinks, c must also be close to a half. So he says, okay, lets figure out c using some sort of an average. So he chooses two weights a and b. The weighted average would then be a times half plus b times the upvote proportion. He chooses those weights in such a fashion that if you have lots of data, one of the weights vanish & the other becomes unity. So the weights matter only if you have too few upvotes & downvotes.

Having figured out the midpoint c, Wilson has to actually figure out the lower bound x and upper bound y. Now he draws a distribution centered at c....ok so at this point you would need to know what is a distribution, and why you would need one, whether that distribution has a skew & whether its homoskedastic & so on...which is stats 101, so I won't go there. But if you've gotten this far, you should be able to atleast see the intuition behind Wilson's procedure.

Re: How Not To Sort By Average Rating

#114

Post by dpritchett »

davidw wrote: post #110
dpritchett wrote: post #75

Earlier quoted context omitted.

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And…

That much was fairly clear on the site. I'm talking about the math itself.

Wilson's 1927 paper is freely available [1]. I can't say I have brushed up on my statistics enough this decade to verify the math but the basics as I read them are as follows:

- You have a normal distribution (bell curve) of data points, in this case quality scores.

- You wish to sort these points based on their respective vote totals. Any given data point has pos positive votes out of n total votes for that item.

- You have a confidence interval, e.g. 95%. This confidence is expressed in terms of the bell curve, so a 95% confidence is within 1.96 standard deviations of the mean [2].

- You have a Ruby function accepting the aforementioned n, pos, and confidence variables and returning a decimal value representing the normalized confidence_interval_lower_bound, that is the quality score that our input data point has a 95% chance of meeting or exceeding.

- Given a set of data points, evaluate the ci_lower_bound for each, and then sort them accordingly. The results will give you a best-guess sorting that accounts for the fact that some data points will have more votes cast for/against them than others.

[1] http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/...

[2] http://en.wikipedia.org/wiki/1.96

Re: How Not To Sort By Average Rating

#115

Post by yariang »

brown9-2 wrote: post #23
yariang wrote: post #5

While it is good to look at these sorts of mathematically rigorous algorithms, I think I would be frustrated if it was used everywhere. Or, well, maybe not me perhaps, but a non technical user. The beauty of the second algorithm for rating products is that it is straightforward. Having never seen it before I can deduce that 5 stars come before 4 stars and more reviews come before fewer. If I want to skip ahead to the…

But how is Simple Stupid in the Amazon case a better output for the user? Do you, as an Amazon shopper, really believe that the item with one 5-star review is a better bet for you than the item with 580 reviews and an average of 4.5-stars?

I don't, but I can intuitively grasp that a 5-star item with 2 reviews is not reliable. Since I understand how the sorting works, I know I have to jump to the 4.5 star items in order and check how many reviews that item has and if it also has a small number then I will jump to the 4 star items.

The point is, I understand the sorting order and can manipulate them if I am not satisfied with what is presented to me. Having a very esoteric algorithm is a risk. Maybe you'll present just what the user really wanted. But if you get it wrong they will be lost to do anything about it. I tend to dislike systems that leave users helpless when something goes wrong.

Re: How Not To Sort By Average Rating

#116

Post by biot »

dxbydt wrote: post #88
edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

>What human can understand it? Lets start with Wilson's midpoint, since that's just high school math. def mid(upvotes:Int, downvotes:Int) = { val total = upvotes+downvotes+0.0 val up = upvotes/total val half = 0.5 val a = total/(4+total) val b = 4/(4+total) a * up + b * half } So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes. It should be…

The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it.

Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably correct. However, they're still going to field questions from merchants saying "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"

Similarly, you may work in the acquisitions department of BigCorp and you need to purchase some equipment. When you prepare your report for Big Boss of your recommendations, how do you explain that you are recommending something which has a 4.5-star average rating instead of the 5-star average rated one?

The ratings may be 100% mathematically sound, but the users of the rating system are not the same as the producers of the rating system. So it's largely irrelevant if the producers made it perfect if the users don't understand what "perfect" means.

Re: How Not To Sort By Average Rating

#117

Post by gcb »

biot wrote: post #116
dxbydt wrote: post #88

Earlier quoted context omitted.

>What human can understand it? Lets start with Wilson's midpoint, since that's just high school math. def mid(upvotes:Int, downvotes:Int) = { val total = upvotes+downvotes+0.0 val up = upvotes/total val half = 0.5 val a = total/(4+total) val b = 4/(4+total) a * up + b * half } So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes. It should be…

The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably c…

the article also misses the human factor. And intent.

for amazon, is it better to show the best product first or to help sell a few of the lacking-rating ones to generate more ratings and work out the uncertainty with real world data instead of crazy math?

Re: How Not To Sort By Average Rating

#118

Post by PenZenMaster »

But shouldn't the solution (formula) be "simply elegant"? eBay seems to be on to something with: positive/positive + negative rating system. The user knows how many data point are in the pool which over comes the one positive rating gets five stars. Much in the same way http://demanddriventech.com/home/solutions/replenishment/ has come up with a "simply elegant" formula for supply chains that is human understandable and effectively solves the problem.

Re: How Not To Sort By Average Rating

#119

Post by dailyllama »

raldi wrote: post #30
edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

I used the wilson score method to generate star ratings off of individual user up/down reviews on a commerce site thirteen years ago.

You can do something very similar with plaintext reviews by doing sentiment analysis to mark them as a yay, a nay, or a no opinion, and generate a similar score.

There are other, more complicated, ways of doing things, but this is fine for somebody who only has a modest knowledge of stat.

Re: How Not To Sort By Average Rating

#120

Post by excuse-me »

excuse-me wrote: post #43

That's why a friend of mine joined the army engineers. As a civil engineer working for a local city he might be involved in a 10year process of approvals to add a freeway on ramp. Where most of his job would be checking that an army of subcontractors were all doing things to code - not that they were doing things well, just to the written requirements In Afghanistan if they want a road or a barrier he basically finds…

EDIT - for some weird reason this cross-posted from another news.y topic. Please ignore it here.
Post reply on HN