Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Ratings Systems Don't Work

goodfil.ms

Post reply on HN
Print view View original

141–150 of 165 posts

Re: Why Ratings Systems Don't Work

#141

Post by graue »

biostat wrote: post #126
dclowd9901 wrote: post #117

Earlier quoted context omitted.

> "Who is rating the raters?" Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own . You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it. This is the only good way of going about this method. The trick is,…

Is Netflix really the only site which does this nearly-braindead machine learning approach? Once you realize that people have different tastes and you know someone's preferences that is the obvious solution. Or is the process of crawling through that much statistical data that expensive that it can only be offered to paying subscribers?

rateyourmusic.com sort of has this, but it's not in the default view. You have to go to an album and click a "View my suggested rating" button, then it whirs for a few seconds before giving you an average of ratings from users with similar taste. It would be much more useful to browse the whole site with those ratings showing, but I get the feeling it's a computationally expensive feature.

Re: Why Ratings Systems Don't Work

#142

Post by Stonewall9093 »

think-large wrote: post #72

You can't rate the rater's because that's all subjective. This is where a nice machine learning algorithm that looks at your scores and find people who have similar rating trends. It would then give weight to other peoples ratings based on your ratings. Thus is born a system in which everyone is happy. The site will get more ratings because people want the algorithm to be better and it needs more data for that. The m…

Hey, check out the site we're working on: www.criticrania.com. We're doing exactly what you mention. You'll need to sign up to see these features, but it's something we're working hard towards perfecting.

Re: Why Ratings Systems Don't Work

#143

Post by Too »

dclowd9901 wrote: post #117
tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

> "Who is rating the raters?" Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own . You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it. This is the only good way of going about this method. The trick is,…

Pandora does something similar. They only have "like" and "dislike" rating. If you like a song they look at other users who liked that song and try to find more songs/bands from the people with your taste. And the other way around for dislike i guess.

It works exceptionally well. You just listen to the stream of incoming songs, you never pick songs yourself. After a good song you click like, after a mediocre song you keep listening, during a bad song you click skip. After a few days you will only get good songs! (with a few exceptions of course) It's like magic, i can't even count how many new bands i found through pandora without any effort.

Too bad it doesn't work outside US anymore. :(

Re: Why Ratings Systems Don't Work

#144

Post by dredmorbius »

karpathy wrote: post #138

Personally, I've consistently found that the best predictor of whether or not I was going to enjoy a movie was the NUMBER of ratings, not the rating itself. This also works for restaurants and other things on sites like Yelp. It almost seems that a movie should come with a simple "recommend!" button that simply counts recommendations. But ratings are a tricky issue and I think they require a more sophisticated mathem…

That metric starts to fail when you've got, say, mediocre products or brands with tons of exposure.

The classic is in "readers choice" reviews of restaurants or eateries. Fast-food franchises dominate? Why? Because the philosophy of such sites is often "majority rules", and the establishment (or brand) with the most votes wins. But there are far more McDonalds or Taco Bells than Jacks Cook Shacks or Trader Vics. Even when the quality of JCS or TV exceeds TB or MD, it's not going to be reflected in the ratings.

Adjustments such as taking a Likert (3-7 point scale) and adjusting reviews based on the number of reviewers, to give both the actual qualitative assessment, and the probable maximal review can help. This is how sites such as Reddit have adjusted their comments/submissions ratings.

The broader and more philosophical problem is that "quality" is not a one-dimensional attribute, interpretation of quality differs among individuals, and "fitness for purpose or task" should be considered when assessing quality as well. McDonalds may very well be appropriate when your goal is a quick, inexpensive meal on the run (a conclusion I'd differ with), while Trader Vics is where you'd head to impress the boss, date, in-laws, or client.

It's a tough problem. It's also one that sees a great many very poor proposed solutions.

Re: Why Ratings Systems Don't Work

#146

Post by batista »

slantyyz wrote: post #114
batista wrote: post #92

Earlier quoted context omitted.

Because "rewatchability" is not just useful to judge how much people like to watch a movie again -- it's also useful to judge how much people like the movie, find it deep, and connect with it (over pure mindless fun). So rewatchability roughly translates to "emotional connection + quality".

For adults, this might be true, but for kids, i'm not so sure. If you've got kids, you know they'll just watch the same flashy animated movies over and over again, even if their opinion of the content is "meh". What would probably happen is that family films, especially animated ones, would have skewed results.

Kids of that age (that watch the same "flash animated movies" over and over again) seldom buy/rent movies themselves.

Plus, they do not normally use online rating systems...

Re: Why Ratings Systems Don't Work

#147

Post by slantyyz »

batista wrote: post #146
slantyyz wrote: post #114

Earlier quoted context omitted.

For adults, this might be true, but for kids, i'm not so sure. If you've got kids, you know they'll just watch the same flashy animated movies over and over again, even if their opinion of the content is "meh". What would probably happen is that family films, especially animated ones, would have skewed results.

Kids of that age (that watch the same "flash animated movies" over and over again) seldom buy/rent movies themselves. Plus, they do not normally use online rating systems...

Good point, but I do see adult-entered reviews at places like Amazon where they write "My kid watches this all the time, bla bla".

Re: Why Ratings Systems Don't Work

#148

Post by SpectralShards »

I really don't think you've tackled the problem at the correct angle here. Anyone can assign two or more axis to rate things on and call it a better rating system than a one dimensional one because it provides more information. Viewers want more information before deciding whether or not a thing (in this case, a movie) is worth their time. While your two dimensional rating system may work for some, this specific system is only justified for people who want information on "quality" and "rewatchability." For example for a person like me, I don't rewatch movies so the "quality" axis offers me way more value and insight compared to the "rewatchability" side.

Re: Why Ratings Systems Don't Work

#149

Post by voyou »

I used Goodfilms briefly, but the rating system is so bad I stopped. The odd thing is that, as this blog post demonstrates, they recognize the problem and then totally fail to solve it. Ratings tend to be fairly bimodal, with people either liking or disliking stuff and not making fine-grained choices. In response to this, the Goodfilms system makes ratings continuous, so that rather than trying to figure out whether a film is 4 or 5 stars, the user now has to figure out whether it's 3.8 stars or 4.6, then compounds the problem by making the user rate on two separate scales with a pretty opaque distinction. So the response to the observation that people's ratings tend to be simplistic is to make the rating system much more complicated; it's pretty much the exact opposite of a solution (I quite like letterboxd's system, which has five-star ratings and also a "like" button, which gives you some level of choice over how fine-grained you want to make your ratings).

Re: Why Ratings Systems Don't Work

#150

Post by CamperBob2 »

CWuestefeld wrote: post #91
podperson wrote: post #45

I definitely go by the 4.5 stars == very good, E.g. when I go to Amazon I don't buy some random product with a 4.5 star review -- I search for a specific product or a specific kind of product and then reject candidates which are lousy. How is that not INCREDIBLY useful? Similarly, who goes to a movie simply based on whether it's good or not. In general, if you create any point rating system people who like a thing wi…

The problem that XKCD gets at is simply translating/scaling the results; the article is solving a different problem. Early on in the Netflix Challenge, I was able to get myself (very briefly) a leaderboard score with nothing more than analyzing every user's ratings; re-centering them by their mean, and re-scaling them according to their standard deviation. The by remembering their translations and scales, I could put…

I was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple

Netflix does have really interesting blind spots. They claim to take ratings seriously, to the point of offering a million dollars for the best rating algorithm. Then, as the GP says, they implement the rating algorithm in a way that renders it completely worthless to any household with more than one viewer.

Netflix does offer us a good demonstration of the failings of absolute technocracy, but it leaves the question of how best to rate movies wide open.

Post reply on HN