I knew I had seen a similar XKCD before. http://xkcd.com/937/
Why Ratings Systems Don't Work
31–40 of 165 posts
Re: Why Ratings Systems Don't Work
#32The problem with ratings are that they are not from a large enough random sample. Rating scores tell me what people that like to rate products think. I don't rate products and in general I suspect that people that are happy with a product don't really care to take the time to go out of their way to tell people about it online (that may change or be changing, I don't know). but one thing is for sure, people that dislike a product WILL go out of their way to tell everyone, thus further shifting the data set to being reviewed by people that are unhappy.
Take a random sample of the true population of the data set (everyone that has seen a movie) and not just the people that logon to rate it.
Re: Why Ratings Systems Don't Work
#33Earlier quoted context omitted.
Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?
The comment I was replying to said This would be no different than having two histograms side by side. And I pointed out that it is not. Oh well I'll answer your question anyway. Because 90% of everything is crap (probably more than that on P2P sites). Re-watching a movie can be like walking in the same park more than once, looking at pleasant things with a sense of recognition. You could put a movie in to suit your…
If I have already seen this movie why would I need ratings of any kind?
Re: Why Ratings Systems Don't Work
#34Earlier quoted context omitted.
The comment I was replying to said This would be no different than having two histograms side by side. And I pointed out that it is not. Oh well I'll answer your question anyway. Because 90% of everything is crap (probably more than that on P2P sites). Re-watching a movie can be like walking in the same park more than once, looking at pleasant things with a sense of recognition. You could put a movie in to suit your…
If I have already seen this movie why would I need ratings of any kind?
What use case are you referring to? I don't know anyone who uses ratings sites after seeing a movie.
Re: Why Ratings Systems Don't Work
#35If you're interested in a more rigorous analysis of the problem, I highly recommend reading the paper "How to Count Thumb-Ups and Thumb-Downs: User-Rating based Ranking of Items from an Axiomatic Perspective":
http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...
It's very accessible for an academic paper.
Re: Why Ratings Systems Don't Work
#36Terrible article. Calling histograms awful, based on nothing more than an opinion. Then trying to conclude that some convoluted scatter plot system makes more sense is laughable. Not to mention, this system is still just a star rating system. This would be no different than having two histograms side by side.. assuming, of course, that you'd even want to rate different aspects of the same thing. I can't even imagine…
There's a pretty clear correlation between the histograms and the scatterplots, too, if you look. If you gave someone those 3 histograms and those 3 scatterplots, I bet they could match them up correctly. Aside- The scatterplots are an awful user interface because of the cognitive effort to interpret them, but perhaps there's a way to present the same information in a usable way.
Please show me any diagram that does not require cognitive effort to interpret.
Re: Why Ratings Systems Don't Work
#37Aren't Bayesian statistics designed to deal with low ratings counts?
I think there was an article about that a while ago on HN.
Re: Why Ratings Systems Don't Work
#38Earlier quoted context omitted.
It's different from having 2 histograms because it lets you distinguish people who thought it was a good movie but not re-watchable, from people who thought it was a good movie and fairly re-watchable, from people who thought it was an OK movie and not re-watchable. With a bunch of histograms you can't correlate the variables.
Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?
> Why would I care about rewatchability?
Speaking as a film buff; this is actually quite a good guide to the sort of movie it is (when combined with quality). If lots of people mark it good quality, but wouldn't watch it again, that implies that you have to be in the right mood for it.
If people mark rewatchability high, even if the quality rating varies, you know it is much more easy going film.
And so on. Combining data points is good :)
Re: Why Ratings Systems Don't Work
#39I'll say histograms are useful for certain things. If you are looking at electronic devices or camera lenses there's the issue that a certain fraction of people get lemons. Some bad reviews are because of that. Other people have unrealistic expectations of the product and give a bad review. A histogram gives some immediate insight into this problem, and then looking at stratified samples of the reviews helps there on…
Some bad reviews on Amazon are from shipping snafus. If you're going to get any useful info, you have to read the reviews.
Re: Why Ratings Systems Don't Work
#40From a statistician's standpoint, ratings systems suck. But, from a consumer standpoint, they are super easy to understand. A scatter plot system makes sense to me, but I would never put it in front of a user. In my opinion, current ratings systems are 80% UX and 20% data. For example, Newegg uses a pretty intuitive system of allowing you to sort a product page by Best Reviews and Most Reviews. In my opinion, this al…
From a consumer standpoint, a single five star review pushing a product to the top of the list is not easy to understand, it's a pain.
The simple answer there relates back to UX, just don't show the stars when there isn't enough data. Set a minimum number of reviews as a baseline so that you don't get the result you mentioned.
If there is a written review component, make a note of the review but don't quantify the value of said review until the minimum threshold is reached.