Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not To Sort By Average Rating

evanmiller.org

Post reply on HN
Print view View original

21–30 of 34 posts

Re: How Not To Sort By Average Rating

#21

Post by gjm11 »

tel wrote: post #17
gjm11 wrote: post #5

The wrong solutions the article describes are certainly wrong. It doesn't really make much of a case that the particular solution it proposes is particularly good. For instance, here's another much simpler formula that avoids the problems described in the article: score = (pos+1)/(pos+neg+2). It's the posterior mean of Pr(random person likes the product), if your prior is uniform on [0,1]. You can adjust this to favo…

If we're going to use a Bayesian treatment, the real sin is throwing away your uncertainty. What I would love to see is a ranking algorithm that provides an elegant, intuitive interface to fuzzy ranking. If we can't be confident of the ordering, why pretend? It's a usability problem, not a statistics problem. Your estimator is slightly more resilient than Amazon's and somewhat less conservative than the author's. It'…

I think it's a huge usability problem, if we're talking about something Amazon-like that's presenting ranked data to an unsophisticated audience. Why pretend? Because it produces something the user might be able to understand, that's why.

(Yes, it would be extremely cool to make an elegant, intuitive interface to a fuzzy ranking. One reason why it would be extremely cool is that it's totally not obvious how it can be done, or even that it can be done.)

I wasn't suggesting that my estimator is better because it's simpler, nor that anyone should care; just pointing out that the author of the article made a huge leap from "here are two things that don't work" to "and here is the specific arbitrary math-heavy thing I think you should do instead".

Re: How Not To Sort By Average Rating

#22

Post by ck2 »

araneae wrote: post #20
ck2 wrote: post #18

I use what reddit uses, works well http://code.reddit.com/browser/r2/r2/lib/db/sorts.py s = score(ups, downs) order = log(max(abs(s), 1), 10) sign = 1 if s > 0 else -1 if s Let me know if you want my attempt at a MySQL translation of that. ps. why is my account ( _ck_ ) so slow when logged in and my comments marked as "dead" ? Was I flagged as a spammer by mistake? Example: http://news.ycombinator.com/item?id=1219475

That's not really appropriate for what this blog post was talking about, which is rating products. It doesn't make sense for products' ratings to be time sensitive. That would make the most recent items have an artificially inflated score.

But you want to control the weight of a rank for an aging product, no?

Let's say you are ranking cellphones that a store sells and someone has a fond memory of a cellphone they used a few years ago so they vote it up. But the phone itself was posted years ago, compared to the newest phones that came out this year. You don't want their vote to have as much weight because of the product age.

Re: How Not To Sort By Average Rating

#23

Post by swombat »

ck2 wrote: post #18

I use what reddit uses, works well http://code.reddit.com/browser/r2/r2/lib/db/sorts.py s = score(ups, downs) order = log(max(abs(s), 1), 10) sign = 1 if s > 0 else -1 if s Let me know if you want my attempt at a MySQL translation of that. ps. why is my account ( _ck_ ) so slow when logged in and my comments marked as "dead" ? Was I flagged as a spammer by mistake? Example: http://news.ycombinator.com/item?id=1219475

ps. why is my account ( _ck_ ) so slow when logged in and my comments marked as "dead" ? Was I flagged as a spammer by mistake?

Looks like it. Looking at your posting history on _ck_, though, I don't see why you'd be auto-deaded. I suggest emailing pg about it.

Re: How Not To Sort By Average Rating

#25

Post by araneae »

ck2 wrote: post #22
araneae wrote: post #20

Earlier quoted context omitted.

That's not really appropriate for what this blog post was talking about, which is rating products. It doesn't make sense for products' ratings to be time sensitive. That would make the most recent items have an artificially inflated score.

But you want to control the weight of a rank for an aging product, no? Let's say you are ranking cellphones that a store sells and someone has a fond memory of a cellphone they used a few years ago so they vote it up. But the phone itself was posted years ago, compared to the newest phones that came out this year. You don't want their vote to have as much weight because of the product age.

You could make an argument for using that for, say, deciding on which products should be featured on a page that's promoting products to consumers.

But for actual product "scores" that people see that's a very bad idea. A book shouldn't have a 4/5 if everyone rated it a 5/5 just because it came out 50 years ago.

Re: How Not To Sort By Average Rating

#26

Post by DougBTX »

Automatt-2 wrote: post #11
Perceval wrote: post #7

Earlier quoted context omitted.

>items with the most reviews are going to rise to the top Yes and no. Anything with a larger N is going to make for greater confidence in the rating. But that rating still depends on the number of positive and negative reviews. So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. The algorithm would pres…

>So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. What I'm saying is that it will rise to the top because even though only 50% of the ratings are positive, it will have many many more positive ratings than the nearest next item. That 2nd item may have a much higher percentage of positive ratings, but…

It seems to work as described in the article:

    require 'rubygems'
    require 'statistics2'

    def ci_lower_bound(pos, n, power)
        if n == 0
            return 0
        end
        z = Statistics2.pnormaldist(1-power/2)
        phat = 1.0*pos/n
        (phat + z*z/(2*n) - z * Math.sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
    end

    puts ci_lower_bound(60, 100, 0.10)   # => 0.517809505446319
    puts ci_lower_bound(500, 1000, 0.10) # => 0.474027691168875
    puts ci_lower_bound(50, 100, 0.10)   # => 0.418847795168265
The top ranked result has 60 positives out of 100, and beats 500 out of 1000 - almost 10 times as many positive votes.

Re: How Not To Sort By Average Rating

#27

Post by Predictor »

gjm11 wrote: post #5

The wrong solutions the article describes are certainly wrong. It doesn't really make much of a case that the particular solution it proposes is particularly good. For instance, here's another much simpler formula that avoids the problems described in the article: score = (pos+1)/(pos+neg+2). It's the posterior mean of Pr(random person likes the product), if your prior is uniform on [0,1]. You can adjust this to favo…

This solution is called a "Laplacian correction", and I agree with your recommendation.

Re: How Not To Sort By Average Rating

#28

Post by tel »

gjm11 wrote: post #21
tel wrote: post #17

Earlier quoted context omitted.

If we're going to use a Bayesian treatment, the real sin is throwing away your uncertainty. What I would love to see is a ranking algorithm that provides an elegant, intuitive interface to fuzzy ranking. If we can't be confident of the ordering, why pretend? It's a usability problem, not a statistics problem. Your estimator is slightly more resilient than Amazon's and somewhat less conservative than the author's. It'…

I think it's a huge usability problem, if we're talking about something Amazon-like that's presenting ranked data to an unsophisticated audience. Why pretend? Because it produces something the user might be able to understand, that's why. (Yes, it would be extremely cool to make an elegant, intuitive interface to a fuzzy ranking. One reason why it would be extremely cool is that it's totally not obvious how it can be…

I agree. I think it's a great problem to motivate a startup on.

Re: How Not To Sort By Average Rating

#30

Post by ntoshev »

gjm11 wrote: post #21
tel wrote: post #17

Earlier quoted context omitted.

If we're going to use a Bayesian treatment, the real sin is throwing away your uncertainty. What I would love to see is a ranking algorithm that provides an elegant, intuitive interface to fuzzy ranking. If we can't be confident of the ordering, why pretend? It's a usability problem, not a statistics problem. Your estimator is slightly more resilient than Amazon's and somewhat less conservative than the author's. It'…

I think it's a huge usability problem, if we're talking about something Amazon-like that's presenting ranked data to an unsophisticated audience. Why pretend? Because it produces something the user might be able to understand, that's why. (Yes, it would be extremely cool to make an elegant, intuitive interface to a fuzzy ranking. One reason why it would be extremely cool is that it's totally not obvious how it can be…

With Bayesian average (http://news.ycombinator.com/item?id=1219154 , http://wiki.answers.com/Q/What_does_true_Bayesian_estimate_m...) you can easily put a slider saying how much the missing votes count (or how much preference do you give to items with more ratings).
Post reply on HN