How Not to Sort by Average Rating (2009) • Hacker News Board

Re: How Not to Sort by Average Rating (2009)

#21

Post by stdbrouw » Thu, Jul 09, 2015, 1:11 PM UTC

vcdimension wrote: post #19

stdbrouw wrote: post #3

Also discussed in Cameron Davidson-Pilon's Bayesian methods for Hackers in the context of Reddit ups/downs: http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabil... For Amazon, though, which is the example in Evan Miller's post, I don't really get why you'd first dichotomize the five-star rating into positive vs. negative and then use Wilson intervals. Just construct a run-of-the-mill 95% confidence interval f…

For better accuracy with small samples you could use the multinomial distribution instead. The covariance matrix for the rating probabilities can be found here for example: http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht... Then the variance for the expected rating can be calculated as a weighted sum of the values in the covariance matrix. These companies really should be hiring statistics consultants in…

I'd prefer to just treat scores as continuous and correct using `t_ppf(.975, n-1)` instead of the normal approximation (1.96) but I suppose working from a multinomial distribution would give pretty similar results.

Re: How Not to Sort by Average Rating (2009)

#22

Post by rjst » Thu, Jul 09, 2015, 1:16 PM UTC

woah wrote: post #4

Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.

I’ve also seen a variation of this that gives different weights to positive and negative votes, the theory being that the likelyhood of you actually voting when you like/don’t like isn’t the same.

Re: How Not to Sort by Average Rating (2009)

#23

Post by vcdimension » Thu, Jul 09, 2015, 1:23 PM UTC

stdbrouw wrote: post #21

vcdimension wrote: post #19

Earlier quoted context omitted.

For better accuracy with small samples you could use the multinomial distribution instead. The covariance matrix for the rating probabilities can be found here for example: http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht... Then the variance for the expected rating can be calculated as a weighted sum of the values in the covariance matrix. These companies really should be hiring statistics consultants in…

I'd prefer to just treat scores as continuous and correct using `t_ppf(.975, n-1)` instead of the normal approximation (1.96) but I suppose working from a multinomial distribution would give pretty similar results.

You're still relying on the central limit theorem (i.e. a reasonable amount of data) : using t instead of z just corrects for the fact that you only have sample variances instead of population variances. However, I suppose it's not unreasonable to assume that the ratings are likely to have a bell shaped distribution (which could be checked), so the normal/t approximation is probably going to be OK.

Re: How Not to Sort by Average Rating (2009)

#24

Post by stdbrouw » Thu, Jul 09, 2015, 1:33 PM UTC

vcdimension wrote: post #23

stdbrouw wrote: post #21

Earlier quoted context omitted.

I'd prefer to just treat scores as continuous and correct using `t_ppf(.975, n-1)` instead of the normal approximation (1.96) but I suppose working from a multinomial distribution would give pretty similar results.

You're still relying on the central limit theorem (i.e. a reasonable amount of data) : using t instead of z just corrects for the fact that you only have sample variances instead of population variances. However, I suppose it's not unreasonable to assume that the ratings are likely to have a bell shaped distribution (which could be checked), so the normal/t approximation is probably going to be OK.

Ah yes, true. Let's call it a bias/variance tradeoff ;-)

Re: How Not to Sort by Average Rating (2009)

#25

Post by danialtz » Thu, Jul 09, 2015, 2:20 PM UTC

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost.

Assume user has/obtains 1000 karma points a month. If the user merely likes a post or not he gives his 1 vote, which cost him 1 karma. If he strongly wants one post up he can spend a maximum of 31 vote on it. This way the minorities will have also extra influence on the voting process.

The requirements is that each user have 1 account, e.g. maybe by some form of payment for 1000 karma to avoid fake voting fraud. Maybe using bitcoins if it picks up to avoid privacy problems.

Do you think this will hinder the workings of sites like reddit?

Re: How Not to Sort by Average Rating (2009)

#26

Post by Cthulhu_ » Thu, Jul 09, 2015, 2:55 PM UTC

danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

> Do you think this will hinder the workings of sites like reddit?

I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity.

It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots and spammers, sure, but no hard cap - or an inhumanly high one at least, bearing in mind that some people are indeed inhuman when it comes to voting/commenting.

Re: How Not to Sort by Average Rating (2009)

#27

Post by billmalarky » Thu, Jul 09, 2015, 2:56 PM UTC

danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

I think sites like reddit just want engagement, even over quality. It's been known forever that reddit's algorithm leaves much to be desired but it's too risky to change.

IE the following (since patched) bug: http://technotes.iangreenleaf.com/posts/2013-12-09-reddits-e...

Re: How Not to Sort by Average Rating (2009)

#28

Post by bwy » Thu, Jul 09, 2015, 3:07 PM UTC

danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

What I always thought was that there really should be some user-based weighting system. Like, if a user upvotes 90% of the things he sees, his upvotes are probably worth less than upvotes by someone who upvotes only 1% of the posts he sees.

Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are probably just lower than the guy with a 1-star average.

The problem is that there are so many other factors here (maybe the 5-star average person only visits (? or just reviews) really good places). Maybe the crazy upvoter just spends more time reading each page on Reddit. These are complicating factors that are hard to predict and if the simple case is working, why try? If there were a simple, clearly better way of voting/rating, it would be done.

Re: How Not to Sort by Average Rating (2009)

#29

Post by Cthulhu_ » Thu, Jul 09, 2015, 3:13 PM UTC

rjst wrote: post #22

woah wrote: post #4

Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.

I’ve also seen a variation of this that gives different weights to positive and negative votes, the theory being that the likelyhood of you actually voting when you like/don’t like isn’t the same.

This is true, especially for things like app store ratings - people are much more likely to leave a rating if they really don't like something. Or if they really do like something for that matter.

Nowadays some apps encourage users more to leave a rating though. I was under the impression that Apple did not approve of those practices, although I might be confused with another rule that stated apps should not suggest 5-star ratings.

The Play Store, iirc, and probably Steam too, have a smart algorithm that display the 'most helpful' reviews at the top, so that as a potential customer you can determine the pros and cons of the app according to other users. Ratings are often very polarized - like up / downvotes.

Re: How Not to Sort by Average Rating (2009)

#30

Post by nzealand » Thu, Jul 09, 2015, 3:14 PM UTC

danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

It might work, if there was a nominal cost to creating an account.

It would be better to take the recommendation engine of netflix. Just throw it on top of reddit and charge a fee for tailored recommendations. Now that is something I would pay money for.