Why Ratings Systems Don't Work • Hacker News Board

Re: Why Ratings Systems Don't Work

#91

Post by CWuestefeld » Wed, Aug 22, 2012, 2:40 PM UTC

podperson wrote: post #45

I definitely go by the 4.5 stars == very good, E.g. when I go to Amazon I don't buy some random product with a 4.5 star review -- I search for a specific product or a specific kind of product and then reject candidates which are lousy. How is that not INCREDIBLY useful? Similarly, who goes to a movie simply based on whether it's good or not. In general, if you create any point rating system people who like a thing wi…

The problem that XKCD gets at is simply translating/scaling the results; the article is solving a different problem.

Early on in the Netflix Challenge, I was able to get myself (very briefly) a leaderboard score with nothing more than analyzing every user's ratings; re-centering them by their mean, and re-scaling them according to their standard deviation. The by remembering their translations and scales, I could put a globally-predicted score back into their own language.

So just some very basic statistics is sufficient to erase much of the bias toward higher numbers, as well as halo effects and the like.

(I was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple)

Re: Why Ratings Systems Don't Work

#92

Post by batista » Wed, Aug 22, 2012, 2:40 PM UTC

guard-of-terra wrote: post #18

sp332 wrote: post #14

Earlier quoted context omitted.

It's different from having 2 histograms because it lets you distinguish people who thought it was a good movie but not re-watchable, from people who thought it was a good movie and fairly re-watchable, from people who thought it was an OK movie and not re-watchable. With a bunch of histograms you can't correlate the variables.

Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?

Because "rewatchability" is not just useful to judge how much people like to watch a movie again -- it's also useful to judge how much people like the movie, find it deep, and connect with it (over pure mindless fun).

So rewatchability roughly translates to "emotional connection + quality".

Re: Why Ratings Systems Don't Work

#93

Post by batista » Wed, Aug 22, 2012, 2:44 PM UTC

guard-of-terra wrote: post #48

sp332 wrote: post #34

Earlier quoted context omitted.

What use case are you referring to? I don't know anyone who uses ratings sites after seeing a movie.

My point exactly. And why would you care about rewatchability if you didn't see the movie once?

Because the rewatchability rating doesn't just tell you how rewatchable a movie is, but also how good and "timeless" it is, i.e mainly how deeply it resonates with people.

This is not about picking a movie to watch again after you have already seen it once: this is about picking a movie you haven't seen but that is so good that other people like to watch it again and again.

Re: Why Ratings Systems Don't Work

#94

Post by erikb » Wed, Aug 22, 2012, 2:58 PM UTC

Already the idea to explain a self-explaining comic was giving me clues about what I would read here. Wasn't disappointed. If one thing of something is bad, it becomes good if you throw 2 of that at your users. Sounds like a solution.

And the comic is also explained now. Thanks.

Re: Why Ratings Systems Don't Work

#95

Post by happimess » Wed, Aug 22, 2012, 3:00 PM UTC

runn1ng wrote: post #59

I still don't understand how is "rewatchability" different from "quality". If the movie is good, I would like to watch it again. What is the difference?

The movie Requiem for a Dream is, to me, a perfect example of a high quality movie that I will not watch twice.

The acting, directing, cinematography, pacing, and sound design are all excellent. However, the film is such a grueling emotional experience, I don't foresee myself sitting down with it again.

Contrast that to, say, Airplane. It's a good movie. Funny. If I'm bored in a hotel room, I'll watch it again. However, it is emphatically not a "great film."

Re: Why Ratings Systems Don't Work

#96

Post by aw3c2 » Wed, Aug 22, 2012, 3:08 PM UTC

This is not helpful at all. They are comparing the one-dimensional 1 to x star rating to something two dimensional.

Re: Why Ratings Systems Don't Work

#97

Post by achterband » Wed, Aug 22, 2012, 3:12 PM UTC

Years ago I created a website for local bands. I also liked to have a rating system and tried an emotion rating system (example: http://achterband.nl/sickboys_and_lowmen/white_buffalo). I think for music and films this is more suitable than giving it a single number. And maybe this applies to other ratings as well.

(Translation of the emotions used: happy, relaxing, surprising, aggressive, sad, explosive)

Re: Why Ratings Systems Don't Work

#98

Post by Stonewall9093 » Wed, Aug 22, 2012, 3:12 PM UTC

Interesting article but it's still arbitrary. I don't think it solves the problem still. What it boils down to is trusting the raters. This is the only way to get by spam, ignorance, and trolling.

I think the solution is knowing who is rating what you're looking at. In the old sense - "Quality over quanitty." I don't need to know what the whole world gave it, just a few people who I've come to trust. We're trying to do this here: www.criticrania.com.

I think this is the only real way around this problem.

Re: Why Ratings Systems Don't Work

#99

Post by espeed » Wed, Aug 22, 2012, 3:14 PM UTC

Randy Farmer goes over this concept in detail in Building Web Reputation Systems (http://shop.oreilly.com/product/9780596159801.do).

Google Tech Talk: http://www.youtube.com/watch?v=Yn7e0J9m6rE

See also: "YouTube: Five Stars Dominate Ratings" (http://youtube-global.blogspot.com/2009/09/five-stars-domina...)

Re: Why Ratings Systems Don't Work

#100

Post by tsahyt » Wed, Aug 22, 2012, 3:15 PM UTC

s_henry_paulson wrote: post #2

Terrible article. Calling histograms awful, based on nothing more than an opinion. Then trying to conclude that some convoluted scatter plot system makes more sense is laughable. Not to mention, this system is still just a star rating system. This would be no different than having two histograms side by side.. assuming, of course, that you'd even want to rate different aspects of the same thing. I can't even imagine…

It's not a terrible article at all. It's a suboptimal solution but the article is pointing out what's wrong with 5-star rating systems. It's labelled as a "response", not as a solution to all our earthly rating problems.