Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

21–30 of 158 posts

Re: How Not to Sort by Average Rating (2009)

#21

Post by fnordian_slip »

nfriedly wrote: post #10
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

I wonder if a system that assigned weights to each individual user's rating based on that user's rating history could help there - if a user always rates products with 5-stars, then another 5-star rating shouldn't have nearly as much weight as one coming from a user that gives a fairly balanced range of ratings. I'm not sure if that would actually work better in practice, but it's at least an interesting idea.

At first your system seemed to me like a solution to bought reviews (mturk and otherwise) and bots.

Then I realized that it would just incentivise bots to add 1-star reviews to random products once their creators figure out this mechanism.

Sometimes these problems make me sad, it could all be so nice and easy if it weren't for these bad actors.

Re: How Not to Sort by Average Rating (2009)

#23

Post by nfriedly »

fnordian_slip wrote: post #21
nfriedly wrote: post #10

Earlier quoted context omitted.

I wonder if a system that assigned weights to each individual user's rating based on that user's rating history could help there - if a user always rates products with 5-stars, then another 5-star rating shouldn't have nearly as much weight as one coming from a user that gives a fairly balanced range of ratings. I'm not sure if that would actually work better in practice, but it's at least an interesting idea.

At first your system seemed to me like a solution to bought reviews (mturk and otherwise) and bots. Then I realized that it would just incentivise bots to add 1-star reviews to random products once their creators figure out this mechanism. Sometimes these problems make me sad, it could all be so nice and easy if it weren't for these bad actors.

Yea, that's the main reason why I'm not sure it could be made to work.

Re: How Not to Sort by Average Rating (2009)

#24

Post by dagw »

nfriedly wrote: post #10
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

I wonder if a system that assigned weights to each individual user's rating based on that user's rating history could help there - if a user always rates products with 5-stars, then another 5-star rating shouldn't have nearly as much weight as one coming from a user that gives a fairly balanced range of ratings. I'm not sure if that would actually work better in practice, but it's at least an interesting idea.

On the other hand most people are only generally moved to bother writing a review for something if it's exceptionally good or exceptionally bad. So someone only giving 5-star ratings could very well be someone who only bothers writing a review if a product is a life changing experience and as such his opinion isn't less valid than that of someone who reviews every little thing he has ever bought.

Re: How Not to Sort by Average Rating (2009)

#26

Post by logicallee »

kstenerud wrote: post #7

That's what's always annoyed me with Amazon's "sort by average rating" setting. I want to see the top 10 or so items by rating to give me a baseline to investigate from, but instead I get page after page of cheap Chinese crap with one 5-star review each from the resident fake reviewer. Worse than useless. Even a simple change like adding a "show only items with a minumum of X reviews" would be a godsend.

> with one 5-star review each from the resident fake reviewer.

don't you think if they switched to this algorithm you would see page after page of cheap crap with 500 5-star reviews each from the 500 resident fake reviewers?

...though now that I've written it like that it does make it obvious that this would be higher a burden on fakers, and cut out some of the products doing this.

Re: How Not to Sort by Average Rating (2009)

#27

Post by onorton »

thanatropism wrote: post #22

Arguably what Urban Dictionary is doing is to weigh by "net favorability" in some sense and quantity of votes. Quantity of votes correlates to relevance, particularly because UD is meant to represent popular usage.

Would a better idea for UD be like the original Facebook "like" system? So you only vote if you think it's relevant and only popular definitions sit at the top.

Re: How Not to Sort by Average Rating (2009)

#28

Post by gknoy »

paulgb wrote: post #17
SamBam wrote: post #9

Earlier quoted context omitted.

That's an interesting hypothesis, but I'd want to see more evidence that "a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star." Surely you can't prove this simply by noticing that there are many 1- and 5-star reviews, as there could be many other reasons for that. One obvious one: people who strongly like or strongly dislike a product are more likely to take the time to review. I…

That's a good alternative hypothesis. It could also be that people's experience with a product really is bimodal: if I order an alarm clock that works as advertised, it is easy to get 5 stars, if it doesn't work at all, it's 1 star. Your explanation works better for why the distribution persists in books and movies though. In any case, I find the mechanism design angle interesting regardless of the behavioral angle :…

I've often hypothesized that most people are more likely to leave a negative review when they are upset, than a positive review when they like something. It certainly holds in my case: I nearly never review things, because it's a giant hassle, so I have to feel really strongly to be willing to spend the time on a review.

Re: How Not to Sort by Average Rating (2009)

#29

Post by eeZah7Ux »

This is computationally very heavy, but, more importantly, for practical purposes you want to have a tunable parameter to balance between sorting by pure rating average and sorting by pure popularity.

Often you also want to give a configurable advantage or handicap to new entries.

Re: How Not to Sort by Average Rating (2009)

#30

Post by CM30 »

gknoy wrote: post #28
paulgb wrote: post #17

Earlier quoted context omitted.

That's a good alternative hypothesis. It could also be that people's experience with a product really is bimodal: if I order an alarm clock that works as advertised, it is easy to get 5 stars, if it doesn't work at all, it's 1 star. Your explanation works better for why the distribution persists in books and movies though. In any case, I find the mechanism design angle interesting regardless of the behavioral angle :…

I've often hypothesized that most people are more likely to leave a negative review when they are upset, than a positive review when they like something. It certainly holds in my case: I nearly never review things, because it's a giant hassle, so I have to feel really strongly to be willing to spend the time on a review.

You're right. It's been proven a bunch of times that people are more likely to leave negative reviews than positive ones, or to remember bad experiences more overall.

Zendesk actually did a survey on this back in 2013:

http://cdn.zendesk.com/resources/whitepapers/Zendesk_WP_Cust...

And American Express found something similar in their Global Customer Service Barometer survey as well:

http://about.americanexpress.com/news/docs/2014x/2014-Global...

Post reply on HN