Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Ratings Systems Don't Work

goodfil.ms

Post reply on HN
Print view View original

121–130 of 165 posts

Re: Why Ratings Systems Don't Work

#121

Post by pixelcort »

Maybe we need something like Pandora's Music Genome Project, but for movies and TV shows.

One of the most interesting features in Pandora is the "Why was this track selected?" action. Imagine something similar where a list of movies and TV shows are presented to you, with sentences for each as to why.

Netflix's recommendations were close, but they still seemed to always focus on one facet at a time, be it a user-predicted rating or a single subcategory of related shows.

Edit: Goodfilms seems to be better in the it tracks two facets at the same time, which does end up creating a diagonal scale from super funny movies you can wath again and again to super serious ones you'll watch once, but that's still not quite like filtering down on tons of facets at the same time.

The closest thing I can think of is the metadata from TV Tropes.

Re: Why Ratings Systems Don't Work

#122

Post by Homunculiheaded »

I'm surprised this article, nor the discussion here doesn't address the main issue here: Just because you use numbers doesn't mean your data is quantitative.

Star scores are an attempt to map a qualitative experience (enjoyment of the film) with some quantitative measure. Which is fine if you just want to get a sense of 'how much' somebody liked a something. If I say I give scotch A a 5 scotch B a 3 and scotch C a 4 then you know that I like the scotch's in A, C, B order. It's a short hand way to express my personal ordering of qualitative experience, just like we use the words: 'good', 'better', 'best'.

The problem is this data is not really numerical, so even basic mathematical operations don't make any sense. When we add 2 heights, 2 masses, 2 speeds etc the result makes sense. But not so with ratings. Even basic difference doesn't make sense, is the difference between 5 and 4 stars the same as between 4 and 3 stars? There is no 'unit' distance in scoring system. So doing any sort of averaging is just going to give you nearly meaningless results.

Re: Why Ratings Systems Don't Work

#123

Post by slantyyz »

ericcholis wrote: post #40
nodata wrote: post #6

Earlier quoted context omitted.

From a consumer standpoint, a single five star review pushing a product to the top of the list is not easy to understand, it's a pain.

The simple answer there relates back to UX, just don't show the stars when there isn't enough data. Set a minimum number of reviews as a baseline so that you don't get the result you mentioned. If there is a written review component, make a note of the review but don't quantify the value of said review until the minimum threshold is reached.

When there are only a handful of reviews, I find myself using "gymnastics rules" and throwing out the best and worst score.

Probably not very scientific though.

Re: Why Ratings Systems Don't Work

#124

Post by ghaff »

zerostar07 wrote: post #58
tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

Many reviewers are biased and judgemental. I prefer to look at the distribution of ratings that amazon shows. Good items have a distribution with one peak, even if it isn't at 5 stars. When ratings create a saddle the product is probably a fluke.

The cute XKCD comic aside, the distribution is also useful because, for certain types of things, a lot of fairly to very negative comments are illuminating even if the average rating is still pretty high. It's not just about polarizing material. If you look, for example, at genre fiction you'll get a lot of fans who give 5s no matter what sort of crap the current book is. But if there are also a notable number of 1s and 2s, that's often a good red flag.

Re: Why Ratings Systems Don't Work

#125

Post by benvanderbeek »

tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

Regarding the unhelpfulness of online reviews, my company has problems with manufacturers/sellers writing 5-star reviews of their own product listings (ASIN's) on Amazon. We've begun (manually) data mining 5-star reviews to identify whether each 5-star-reviewer has any other reviews (or wish list, to indicate the possibility of a real user account), then calculating the % of reviews written by no-history user accounts. Of the ASIN's we've assessed, the gut-level-doesn't-seem-like-heavy-review-fraud listings can be in the 6% range, whereas the looks-like-review-fraud ASIN's are above 20%. We're working with Amazon to identify and penalize these manufacturers/sellers, but internally at Amazon the Seller Performance team is separate from their Community (user review) team, so it presents a challenge. Also hard for them to separate valid complaints from sour grapes complaints.

Re: Why Ratings Systems Don't Work

#126

Post by biostat »

dclowd9901 wrote: post #117
tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

> "Who is rating the raters?" Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own . You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it. This is the only good way of going about this method. The trick is,…

Is Netflix really the only site which does this nearly-braindead machine learning approach?

Once you realize that people have different tastes and you know someone's preferences that is the obvious solution. Or is the process of crawling through that much statistical data that expensive that it can only be offered to paying subscribers?

Re: Why Ratings Systems Don't Work

#127

Post by bstpierre »

saucerful wrote: post #90

Surprised to see no mention of the website Criticker. Been using it for a few years now (ever since I cancelled Netflix and missed the recommendation engine). Criticker's rating system is out of 100 points but for each user it scales ratings into tiers (deciles) 1-10. So for someone like me who watches lots of movies that I sort of know I'm gonna like (thanks to Criticker!), most of my ratings end up in the 70 to 100…

sigh

A butterfly flaps its wings, xkcd puts up a comic on ratings, someone piggybacks on the comic, it makes the front page of HN, you wander by and mention criticker, a bunch of geeks pile onto the site to check it out... and it ends up crashy for a while.

Cool site, thanks for mentioning it. From what I saw before it went down (too many mysql connections?), it even looks like I can export my ratings.

Re: Why Ratings Systems Don't Work

#129

Post by SilasX »

thebigshane wrote: post #106
bgilroy26 wrote: post #88

Earlier quoted context omitted.

Re: Your watchmen quote I think Amazon's "Was this rating helpful (Yes/No)?" provides a good filter for ratings. A lot of mindlessly negative reviews get filtered out by the users who come along afterwards and rate the rating in their own self-interest.

That can be easily gamed as well. If you want to boost a rating for a book, mark all of the negative comments as not helpful... In fact, I see that happen on Amazon and Newegg a lot.

They have a solution for that -- on Amazon (at least) to filter for the most helpful unfavorable reviews.

Re: Why Ratings Systems Don't Work

#130

Post by lunchbox »

tensafefrogs wrote: post #71
tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

> The problem with any commercial website with ratings (Amazon, Yelp, etc., etc.) is that there is HUGE incentive to game the ratings. While I agree that this is a problem, I think a bigger problem is a simple matter of scale: Amazon is huge, and many people buy things, but they don't split reviews and ratings by what kind of person is rating them. If they wanted to make an improvement, why not show me only ratings a…

Like Netflix
Post reply on HN