Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

41–50 of 62 posts

Re: How Not to Sort by Average Rating (2009)

#41

Post by dredmorbius »

stdbrouw wrote: post #15
Houshalter wrote: post #14

There is a much simpler and elegant method. Just rank posts by their probability of getting an upvote. This is just (upvotes+1)/(upvotes+downvotess+2).

This gives an advantage to new posts for which the probability is much more uncertain: it's easier to get 1 upvote and 0 downvotes (rank 2/3) than to get 1999 upvotes and 999 downvotes (also rank 2/3). Maybe that's what you want, but the post is exactly about those cases when this is not what you want.

New posts frequently start with a disadvantage though in existing systems. Temporarily biasing them favourably increases odds of any moderation. Alternatively you could present them only to a subset of readership. I've suggested this as a solution to HN's new submissions queue problem.

Increase the presentation as ratings increase.

Re: How Not to Sort by Average Rating (2009)

#42

Post by cgearhart »

discardorama wrote: post #39

How well does this work when you don't have a binary (+/-) rating system, but a multi-valued one (1 - 5 stars) ?

See the discussion elsewhere in this thread for confidence intervals in multinomial distributions. https://news.ycombinator.com/reply?id=9856607&goto=item%3Fid...

Re: How Not to Sort by Average Rating (2009)

#44

Post by jmilloy »

danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

I find this to be a strange comment. Quadratic voting is much better if we have to reach a consensus and want to protect minorities. So it has it's application (and is woefully under-used). But surely it's not always the best method. I think often you want the most popular comment to be at the top of the list, because it's what appeals to the most people.

Secondly, one of the big problems that quadratic voting addresses is when an ambivalent majority outweighs a minority simply because everyone votes. But not everyone has to vote on every comment on a site like reddit, so the ambivalent majority is already ignored.

Re: How Not to Sort by Average Rating (2009)

#45

Post by marcosdumay »

bwy wrote: post #28
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

What I always thought was that there really should be some user-based weighting system. Like, if a user upvotes 90% of the things he sees, his upvotes are probably worth less than upvotes by someone who upvotes only 1% of the posts he sees. Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are…

Yes, Bayesian weighting. I feel that every kind of pool needs Bayesian correcting for calculating how much information does an opinion really carry.

The problem of people only visiting isn't really a problem. If a person only visits good places, he'll be perfectly able to differentiate those good places from each other, and can say one of them is bad. Yet, somebody that visits good and bad places has a saying on what places are good, and what ones are bad.

Re: How Not to Sort by Average Rating (2009)

#46

Post by timon999 »

bwy wrote: post #28
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

What I always thought was that there really should be some user-based weighting system. Like, if a user upvotes 90% of the things he sees, his upvotes are probably worth less than upvotes by someone who upvotes only 1% of the posts he sees. Same thing applies to things like Yelp reviews. Maybe a user with close to a 5-star lifetime rating average should have his reviews "renormalized" to 3's because his standards are…

But this encourages donwnvoting, which is something you definitely do not want.

Re: How Not to Sort by Average Rating (2009)

#47

Post by anon4327733 »

First two points are great, but why then we see this:

"Given the ratings I have, there is a 95% chance that the "real" fraction of positive ratings is at least what?"

What normal person thinks in terms of confidence intervals?

The obvious answer is people want the product with the highest "real" rating. That is the rating the product would get if it had arbitrary many ratings.

To get this you just find the mean of your posterior probability distribution. For just positive and negative reviews thats basically (positive+a)/(total+b) where a and b depend on your prior.

His proposal would mean that a product with zero reviews would be rated below a product with 1 positive review. This may deal with spam and vote manipulation since things with less information are penalized more but that is a separate issue.

Re: How Not to Sort by Average Rating (2009)

#48

Post by brudgers »

Cthulhu_ wrote: post #26
danialtz wrote: post #25

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 ), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user h…

> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots…

Beyond the concerns you raise, vote caps etc., inhibit problematic behaviors mechanistically rather than via better social structures. Limiting the number of posts a person can make per day is one way of addressing potential flame wars when the underlying problem is that flaming is deemed acceptable behavior [and consequently such sites attracts people who engage in flaming and encourage some people to post in ways that get all their digs in at once].

The alternative of building community norms is harder. Mechanism can support it, but success depends on the over-arching social structure.

Re: How Not to Sort by Average Rating (2009)

#49

Post by brudgers »

squeaky-clean wrote: post #37
Cthulhu_ wrote: post #26

Earlier quoted context omitted.

> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots…

I agree that showing the user that they are expending some sort of resource by voting will cause them to act differently or be more conservative with it. An idea I've had is handling this behind the scenes by weighing the votes accordingly. Let's say each user only has 1,000 "points" but infinite votes. Votes 1,000 times? Each vote == 1 point. Just vote once? That vote is worth 1000x as much as the other guy. Spam a…

If people are upvoting "the right things" then limiting votes is counter-productive. If people are upvoting cat-pictures, then the problem is that the site is not selecting the right group of users and better filters are needed so that the community has the desired structure.

Re: How Not to Sort by Average Rating (2009)

#50

Post by jrochkind1 »

gsteinb88 wrote: post #18
jrochkind1 wrote: post #17

Interestingly, I _think_ the Reddit algorithm basically makes this mistake too -- although embedded in a more complicated algorithm that combines with 'newest first' altered by positives minus negatives. I don't think the HN algorithm is public, but wouldn't be surprised if it does the same. Perhaps the generally much smaller number of 'votes' on a HN/reddit post makes it less significant.

For posts, I'm not sure what the algorithm is (I think it's deliberately more complicated, and has to take into account time of posting?), but after this article [the op] was written, reddit implemented the method for comments, as explained by Randall Munroe: http://www.redditblog.com/2009/10/reddits-new-comment-sortin... You only get this ranking method if you sort the comments by 'best' though

Not the default 'top' though, I think.
Post reply on HN