Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Ratings Systems Don't Work

goodfil.ms

Post reply on HN
Print view View original

131–140 of 165 posts

Re: Why Ratings Systems Don't Work

#131

Post by awakeasleep »

CodeMage wrote: post #41

Scatter plots are definitely more informative, once one gives them a couple of minutes to get used to them. However, I think you're shooting at the wrong target and your solution would exacerbate the root problem: bias. The first time I really noticed the problem was when I published my own Flash game on Kongregate and started paying closer attention to the ratings. That led me to examine my own rating habits and I c…

I've noticed the same problem, where people tend to rate a 5 or a 1.

As a benefit, I've noticed I can usually find the best reviews on Amazon by looking for 3 star ratings, and to a lesser extend 4 and 2 stars. People who rate a 3 have looked at pros and cons of the product, and generally compare to similar goods. 3 star reviews usually provide FAR more information than glowing or glowering 4 and 1's.

Re: Why Ratings Systems Don't Work

#132

Post by swah »

guard-of-terra wrote: post #18
sp332 wrote: post #14

Earlier quoted context omitted.

It's different from having 2 histograms because it lets you distinguish people who thought it was a good movie but not re-watchable, from people who thought it was a good movie and fairly re-watchable, from people who thought it was an OK movie and not re-watchable. With a bunch of histograms you can't correlate the variables.

Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?

You're telling me its stupid to choose the 100% chance of some known amount of fun rewatching an old series than to bet compared to watching that latest Jennifer Aniston flick?

Re: Why Ratings Systems Don't Work

#133

Post by tolos »

CodeMage wrote: post #41

Scatter plots are definitely more informative, once one gives them a couple of minutes to get used to them. However, I think you're shooting at the wrong target and your solution would exacerbate the root problem: bias. The first time I really noticed the problem was when I published my own Flash game on Kongregate and started paying closer attention to the ratings. That led me to examine my own rating habits and I c…

Similar to what you said, where YouTube found most people only voted 5 stars, with 1 and 4 stars being used much less frequently: http://youtube-global.blogspot.com/2009/09/five-stars-domina...

Re: Why Ratings Systems Don't Work

#134

Post by sojong »

While this may not be applicable to goodfilms, a 2-dimensional system is harder to brand. As in, "Two thumbs up," or "90% on Rotten Tomatoes." There's value in a single understandable number.

I wonder if there's a measurable value for the system. I care about discovery, so I want a site that can recommend me movies that I wouldn't normally think about. How many more movies would this system recommend to me?

A lot of times, I don't care too much about accuracy as long as the system isn't too far off. This is simply because the cost of an inaccurate recommendation isn't too high when I can stream it on Netflix.

I like the idea of providing more data for people to make more accurate assessments, but I don't necessarily believe optimizing for accuracy optimizes the value provided to the user.

Re: Why Ratings Systems Don't Work

#135

Post by mistercow »

dbecker wrote: post #66
abrookewood wrote: post #16

Earlier quoted context omitted.

I'm with Henry ... the scatterplot is fundamentally no different to two histograms side-by-side (though it is marginally easier to read).

Though a scatterplot conveys less information (e.g. the correlations between the two axes), I think it takes longer to process. It also takes more screen real estate than a pair of histograms.

>Though a scatterplot conveys less information

A scatterplot conveys objectively more information.

Re: Why Ratings Systems Don't Work

#136

Post by sheraz »

username3 wrote: post #82
tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

Relative ratings are more useful. Everyone uses their own scale, but their ratings are relative to the constant movie. I want to see how people rated a movie relative to other movies I've watched. N people rated this better than X movie, but less than Y movie. Ranking movies can be easy. Show 5 movie posters instead of 5 stars or have an auto-complete field for this movie is up there with: .

Anyone interested in relative ratings should look at Dan Areily's[1] Predictably Irrational. He is an economist who writes about behavioral economics and decision making.

I've often thought about some start-up ideas around relative ratings, and this book was the reason

[1] - http://danariely.com/

Re: Why Ratings Systems Don't Work

#137

Post by kragen »

Overview of past discussions:

"How not to sort by average rating" (2009): https://news.ycombinator.com/item?id=3792627 For thumbs-up/thumbs-down systems, suggests using the lower bound of a Wilson confidence interval for a Bernoulli distribution, which is what Reddit does now. Convincingly refuted by How to Count Thumb-Ups and Thumb-Downs: User-Rating based Ranking of Items from an Axiomatic Perspective, http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... by Dell Zhang et al., which argues for simple smoothing with a Dirichlet prior (i.e. (upvotes + x) ÷ (upvotes + x + downvotes + y)), which was also suggested by several people in the comments.

In 2010, William Morgan wrote http://masanjin.net/blog/how-to-rank-products-based-on-user-... partly in response, applying Bayesian statistics to the problem of ranking things rated using 5-star rating systems.

Perhaps related: HotOrNot started out displaying the mean of the rankings as the rating of each photo (after you clicked on it). But they found that there was a gradual drift down in ratings: they started with around 1-5 (out of a theoretical max of 10), then ended up around 1-3, etc., with the predictable damaging effects on egos, people's willingness to post their photos, and the information content of the ratings. The solution they adopted was to display not the mean of ratings but the percentile: a photo rated higher than 76% of other photos would have its "average" displayed as "7.6", even if the mean was 4.5. This trained the users to flatten the histogram!

http://www.nashcoding.com/2011/10/28/hackernews-needs-honeyp... suggested that fake "products" to attract ratings could distinguish intelligent ratings from unintelligent ones. Although written about thumbs-up/down systems, it applies to multi-star systems as well.

Re: Why Ratings Systems Don't Work

#138

Post by karpathy »

Personally, I've consistently found that the best predictor of whether or not I was going to enjoy a movie was the NUMBER of ratings, not the rating itself. This also works for restaurants and other things on sites like Yelp. It almost seems that a movie should come with a simple "recommend!" button that simply counts recommendations.

But ratings are a tricky issue and I think they require a more sophisticated mathematical treatment and modeling if one wants to get it right, not just a few histograms that treat all people equal.

There are a few modeling challenges that come to mind: For example, people disagree on quality of movies based on their taste. This could be modeled as a latent variable that must be inferred for every person in some graphical model. Another example of a relevant variable would be person's rating habits: some people rate movies 5 or 1, some people have a gaussian rating centered at some value. These should be explicitly modeled and normalized. Every rating could ideally be used to make a stochastic gradient update to the weights of the network, and since we are dealing with very sparse data, strong priors and Bayesian treatment seems appropriate. Ratings could then be personalized through an inference process on the graph.

Has anyone heard of a more sophisticated model like this, or any efforts in this direction? I'd like to see more math, modeling and machine learning and less silly counting methods.

Re: Why Ratings Systems Don't Work

#139

Post by goggles99 »

Rewatchability and quality? Are you kidding me?

How do I glean from this wither of not I will like the movie?

A High quality movie can be terrible and a low quality movie can be great. What does quality mean? Does that mean they had good special effects and angles? Does that mean the color is true and the acting was great? How does this equate to me liking the movie?

Rewatchability? Many of my favorite movies I would not watch again (Lord of the rings) because they were so long. There are also so many movies that I want to see in the future that I will choose to watch one of them instead of re-watching one that I have already watched. Re-watching is something that people do less and less as they get older (kids and teenagers maybe do it) but adults (your target audience) not so much.

Also those scatter charts mean the same to my brain as the histograms that you are blasting so I would stick with what people are already used to (the histograms). You are not doing anyone any favors by changing the presentation of the same data. People are not stupid. They will see both equally in most cases but they will prefer the familiarity of the histograms.

Trying to be different is not always the best thing to do. As many have already mentioned. Use machine learning to augment ratings (like Netflix).

Good luck

Re: Why Ratings Systems Don't Work

#140

Post by godDLL »

Homunculiheaded wrote: post #122

I'm surprised this article, nor the discussion here doesn't address the main issue here: Just because you use numbers doesn't mean your data is quantitative. Star scores are an attempt to map a qualitative experience (enjoyment of the film) with some quantitative measure. Which is fine if you just want to get a sense of 'how much' somebody liked a something. If I say I give scotch A a 5 scotch B a 3 and scotch C a 4…

Ah, the top-sense comment is the third one from the top. :)

A store that is also a movie theatre could do away with numeric representations by just watching what the users are doing with it's content. Things like "did they finish watching the movie?" or "did they get through the whole thing in one sitting?" could be helpful. Not to mention you can actually see if those titles were being watched again or not.

But there's still the problem of how to communicate the findings to the user, or formulate them.

Post reply on HN