Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not To Sort By Average Rating

evanmiller.org

Post reply on HN
Print view View original

131–140 of 159 posts

Re: How Not To Sort By Average Rating

#131

Post by ealloc »

davidw wrote: post #31
imajes wrote: post #18

Earlier quoted context omitted.

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regula…

> It just requires better educators to explain it, with analogies and metaphor. So... anyone want to take a stab at explaining that equation to those of us who don't really get it?

Say the 'acutal' rating for an item is p. (ie, if you got an infinite number of people voting, the ratio of upvotes to total votes is p).

Now say your users vote, and they upvote with that probability p. The number of upvotes k you get out of n votes will follow a binomial distribution B(k;n,p). The binomial distribution has mean np and stdev sqrt(n p(1-p)), and is very close to gaussian in shape. Since the stdev is a rough measure of the 'width' of the distribution, common way to describe the error is (mean +/- lambda*stdev), where you can tune lambda to your desire. If you increase lambda you get a wider confidence interval, and therefore more certainty that a measurement will be within that confidence interval.

Now, say you measure k upvotes out of n votes. You can divide by n to get p0, your estimated rating based on those votes.

An easy estimate for the error of this measurement is to assume that p0 is approximately correct and equal to p. Then the expected number of upvotes would be n p0 with stdev(n p0(1-p0)). Divide this by n to get the fraction of upvotes, to give a final estimate for p of p0 +- lambda sqrt(p0(1-p0)/n)

Now, your estimate (p0) of p is not quite right, and therefore your estimate of the error (which depends on p0) is not quite right either. The wilson score attempts to correct for that. We don't know p, but imagine if we did, we would expect any measurement p0 to be in the range p +- lambda sqrt(p(1-p)/n). That is, we expect

        abs(p - p0)
If you solve this equation for p in terms of p0, you get a formula given in the article, ie, the confidence limits for p given p0.

Re: How Not To Sort By Average Rating

#132

Post by lwat »

biot wrote: post #116
dxbydt wrote: post #88

Earlier quoted context omitted.

>What human can understand it? Lets start with Wilson's midpoint, since that's just high school math. def mid(upvotes:Int, downvotes:Int) = { val total = upvotes+downvotes+0.0 val up = upvotes/total val half = 0.5 val a = total/(4+total) val b = 4/(4+total) a * up + b * half } So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes. It should be…

The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably c…

> "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"

'Because you have fewer votes than your competitor'

I don't think it's that hard to understand.

Re: How Not To Sort By Average Rating

#133

Post by Eliezer »

moultano wrote: post #128
Eliezer wrote: post #105

Earlier quoted context omitted.

I read the article, went "The ^&#$?" and came here to post this, more or less. It's amazing the $&#! people will get up to when they don't know Bayesian statistics.

That really isn't fair to what this is doing. If a bayesian wanted to lower bound the distribution, they would be using exactly the same formula, but with an explicit prior. The typical bayesian solution to this of the dirichlet prior has a significantly different convergence rate, 1/n vs 1/sqrt(n) so the resulting ordering is substantially different as well, and I would argue worse for this application. The method o…

If you think most comments are bad, and you have an explicit prior that reflects this, then your ranking will update to "Hey, the comment was actually good!" at the optimal speed - no more, no less - given the rate of incoming ratings. Why would any other rate of updating be better?

Re: How Not To Sort By Average Rating

#134

Post by moultano »

Eliezer wrote: post #133
moultano wrote: post #128

Earlier quoted context omitted.

That really isn't fair to what this is doing. If a bayesian wanted to lower bound the distribution, they would be using exactly the same formula, but with an explicit prior. The typical bayesian solution to this of the dirichlet prior has a significantly different convergence rate, 1/n vs 1/sqrt(n) so the resulting ordering is substantially different as well, and I would argue worse for this application. The method o…

If you think most comments are bad, and you have an explicit prior that reflects this, then your ranking will update to "Hey, the comment was actually good!" at the optimal speed - no more, no less - given the rate of incoming ratings. Why would any other rate of updating be better?

Because there's a high variance on your estimate, so in a lot of circumstances just by variance you'll show users bad comments on top most of the time, rather than showing them the good comment that you are certain of.

In this context, the pessimistic prior is more of a hack than the confidence interval, because your mental model isn't that most comments suck. Your mental model is that most of them are ok. The pessimistic prior is just there to say, if 1000 people confirm that a comment is ok, we should show that over a comment for which we don't know whether it's ok or not yet, and this reasoning is much better modeled by a lower bound.

Re: How Not To Sort By Average Rating

#135

Post by walexander »

raldi wrote: post #30
edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

I actually find issues with the "best" comment sorting, in some cases. I think it generally is the "best" sort, but in very popular posts it tends to break down.I often find comments with 5x or more upvotes buried down below the top few comments.

I think as comments get higher to the top, people start voting them down more, which might not be the cases for comments which are rising (theory: people reading further down in the comment section are possibly more thoughtful and more likely to upvote, while at the top the ADHD crowd might be more likely to knee jerk downvote?).

In any case, i've recently changed my default sort to "top" and feel it's an improvement.

Re: How Not To Sort By Average Rating

#136

Post by celer »

mistercow wrote: post #29
NathanRice wrote: post #22

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer. Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item.…

> this is one of those cases where a Bayesian treatment is conceptually much clearer Is there any other sort of case?

I am a beginning stats student. I took the AP class in high school, and have been doing independent work since. Predicting voters, playing with data sets, using a bit of Python to try to make general solutions, and other non serious but non trivial actions. Bayesian solutions have always required more work from me, even when they seemed the optimal way to solve a problem. I still don't entirely understand how to work with continuous distributions of hypotheses, partially because I lack the deep mathematical intuition at the moment. I don't blame the math, but I do think that much of the time the Bayesian treatment can be more challenging. I suspect that this is why non-Bayesian techniques remain so popular.

Re: How Not To Sort By Average Rating

#137

Post by esrauch »

gliese1337 wrote: post #122
biot wrote: post #116

Earlier quoted context omitted.

The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably c…

They'll only have to field those questions if they display the plain averages despite not sorting on them. If we've established that average rating is the wrong metric, why would you even show it to the user/merchant/whoever? Display the same metric that you sort by, and then the merchants will be perfectly happy with higher-rated items showing up higher on the list, and Big Boss will be happy that you've chosen the…

I really don't think that solves the problem; you look at product X, it has 0 reviews, you give it a 5 star, now it shows 1 rating and 3.7 stars?

Re: How Not To Sort By Average Rating

#138

Post by esrauch »

edash wrote: post #15
jwr wrote: post #14

I implemented this in a rating system once. Got multiple bug reports, people complained that the system calculates averages wrong, because there are two ratings and the average is obviously not the number they are seeing.

Why show the second figure to the user? They don't need to see the calculation you made to determine the sort. Just sort the objects in the order determined by this formula and only show the ratings given by users in the interface.

This is mildly amusing because above in this thread the exact opposite comment/response happened; someone said they used it for sorting but showed the mean next to items, and got bug reports that 4.5 was sorted above 5.0. The reply was that he should have just exposed the numbers that they are sorting by as the visible rating in the interface.

Re: How Not To Sort By Average Rating

#139

Post by esrauch »

ajross wrote: post #65

It's well explained, informally. The giant equation sitting there without clearly defined parameters is mostly just showing off though. The final "QED" solution that you put at the end of a paper is not the proper form to introduce a concept. But... so what? Amazon and Urban Dictionary are hardly failing in the market due to their "incorrect" score sorting. The whole problem is a heuristic, it's not amenable to rigor…

It's been fine in those contexts, but imagine if HN ranked +1/-0 above +100/-1
Post reply on HN