How Not to Sort by Average Rating (2009)
evanmiller.org
How Not to Sort by Average Rating (2009)
1–10 of 158 posts
Re: How Not to Sort by Average Rating (2009)
#2Read this article before and I really liked how to the point it is. More than anything, can I just say how infuriating Amazon's rating system is?
Re: How Not to Sort by Average Rating (2009)
#3Chris Stucchio and Evan Miller have amazing statistics blogs.
Re: How Not to Sort by Average Rating (2009)
#4Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave.
Median ratings are "honest" in this sense, as long as ties are broken arbitrarily rather than by averaging. Math challenge: is there a way of combining the desirable properties mentioned in the post with the property of honesty? I suspect there is but I haven't tried it.
Re: How Not to Sort by Average Rating (2009)
#5Implemented this after discovering it via https://www.gamasutra.com/blogs/LarsDoucet/20141006/227162/F...
Works amazingly well and so easy to calculate vs say the way IMDb rates things.
Re: How Not to Sort by Average Rating (2009)
#6Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…
>A look at rating distributions shows that this is in fact how many people behave.
This is really interesting, do you have anything where I could read more about it?
Re: How Not to Sort by Average Rating (2009)
#7That's what's always annoyed me with Amazon's "sort by average rating" setting. I want to see the top 10 or so items by rating to give me a baseline to investigate from, but instead I get page after page of cheap Chinese crap with one 5-star review each from the resident fake reviewer.
Worse than useless.
Even a simple change like adding a "show only items with a minumum of X reviews" would be a godsend.
Re: How Not to Sort by Average Rating (2009)
#8That's what's always annoyed me with Amazon's "sort by average rating" setting. I want to see the top 10 or so items by rating to give me a baseline to investigate from, but instead I get page after page of cheap Chinese crap with one 5-star review each from the resident fake reviewer. Worse than useless. Even a simple change like adding a "show only items with a minumum of X reviews" would be a godsend.
What's crazy is everyone knows Amazon's ranking is crap, except apparently Amazon - and it's been crappy in the same way for 10 years.
Re: How Not to Sort by Average Rating (2009)
#9Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…
That's an interesting hypothesis, but I'd want to see more evidence that "a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star."
Surely you can't prove this simply by noticing that there are many 1- and 5-star reviews, as there could be many other reasons for that. One obvious one: people who strongly like or strongly dislike a product are more likely to take the time to review. I personally have never felt the need to review something if I felt "meh" about it.
One sample study might be to see how people's ratings change if they have a chance to see the average rating first or not, but that would be a tricky study as you'd need to get people to buy something without seeing the ratings.
Re: How Not to Sort by Average Rating (2009)
#10Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…
I wonder if a system that assigned weights to each individual user's rating based on that user's rating history could help there - if a user always rates products with 5-stars, then another 5-star rating shouldn't have nearly as much weight as one coming from a user that gives a fairly balanced range of ratings.
I'm not sure if that would actually work better in practice, but it's at least an interesting idea.