Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

31–40 of 62 posts

Re: How Not to Sort by Average Rating (2009)

#31

Post by rurban »

I'm ranking movies by critics ratings. Most of them have too low numbers, and thus naive bayesian ranking by avg does not work. IMDB gets away with it, but I cannot. And you should be able to see a good preview of the expected ranking even with low numbers.

So you need to check the confidence interval with Wilson, but you also need to check the quality of the reviewer. There are some in the 90%, but there are also often outliers, i.e. extremities. Mostly french btw.

I updated the c and perl versions, compiled and pure perl here: https://github.com/rurban/confidence_interval

Re: How Not to Sort by Average Rating (2009)

#32

Post by joshuapants »

bwy wrote: post #28
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

What I always thought was that there really should be some user-based weighting system. Like, if a user upvotes 90% of the things he sees, his upvotes are probably worth less than upvotes by someone who upvotes only 1% of the posts he sees. Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are…

> Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are probably just lower than the guy with a 1-star average.

Another problem is the perception of star ratings. It seems like 5 (maybe 4 also) is the only "positive" rating for many people. Anything less and it might as well be a 1.

Re: How Not to Sort by Average Rating (2009)

#33

Post by lmm »

Cthulhu_ wrote: post #26
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots…

> I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity.

It's worth pointing out that Slashdot has run successfully for decades with users getting limited moderation points.

Re: How Not to Sort by Average Rating (2009)

#34

Post by dredmorbius »

lmm wrote: post #33
Cthulhu_ wrote: post #26

Earlier quoted context omitted.

> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots…

> I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's worth pointing out that Slashdot has run successfully for decades with users getting limited moderation points.

Slashdot's moderation applies to comments only, not stories, and leaves a great deal of good discussion on the floor.

And other issues: https://news.ycombinator.com/item?id=9854240

Re: How Not to Sort by Average Rating (2009)

#35

Post by [deleted] »

Cthulhu_ wrote: post #26
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots…

[deleted]

Re: How Not to Sort by Average Rating (2009)

#37

Post by squeaky-clean »

Cthulhu_ wrote: post #26
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots…

I agree that showing the user that they are expending some sort of resource by voting will cause them to act differently or be more conservative with it.

An idea I've had is handling this behind the scenes by weighing the votes accordingly. Let's say each user only has 1,000 "points" but infinite votes. Votes 1,000 times? Each vote == 1 point. Just vote once? That vote is worth 1000x as much as the other guy. Spam a million votes on everything, and each one is basically worthless.

Of course, handling it exactly like that is way too simple, it would need to be balanced more carefully than that. In that naive example, someone who reddits for 5 minutes per month and votes once has as much power as someone online 24/7. Users would also be able to figure out how the system works if scores are visible in some way.

The specific use I played with this for (but never got around to finishing the project) was for movie reviews. There are people who will rate 50% of moves 5-stars and 50% of movies 1-star, but never in between. And some people who are much more selective and may give a 5-star rating to only a few movies ever. A 5-star rating from both of these users should not be valued the same.

Re: How Not to Sort by Average Rating (2009)

#38

Post by bwy »

joshuapants wrote: post #32
bwy wrote: post #28

Earlier quoted context omitted.

What I always thought was that there really should be some user-based weighting system. Like, if a user upvotes 90% of the things he sees, his upvotes are probably worth less than upvotes by someone who upvotes only 1% of the posts he sees. Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are…

> Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are probably just lower than the guy with a 1-star average. Another problem is the perception of star ratings. It seems like 5 (maybe 4 also) is the only "positive" rating for many people. Anything less and it might as well be a 1.

Of course, there's many areas that I didn't even mention. Another thing in the same vein for upvotes that I sometimes think about is, what is the meaning of upvoting? Does it mean, "I like this," or can it also mean "I think this submission should be higher?" Maybe I think too much but I've refrained from upvoting posts I like because I don't think they should be higher than their current position.

Re: How Not to Sort by Average Rating (2009)

#40

Post by jmilloy »

bwy wrote: post #28
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

What I always thought was that there really should be some user-based weighting system. Like, if a user upvotes 90% of the things he sees, his upvotes are probably worth less than upvotes by someone who upvotes only 1% of the posts he sees. Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are…

Criticker does this well. But as you say, I try not to waste time watching movies I don't think I will like, so there is a good reason my ratings are biased towards the high end.
Post reply on HN