Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

91–100 of 158 posts

Re: How Not to Sort by Average Rating (2009)

#91

Post by ehhnetfliz »

hyperpape wrote: post #15
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…

Ehh, netflix switched to that. It's even less useful now: there's no way to indicate you really like a show vs it's not terrible; this means your taste approximately correlates with abailable content, not content you prefer.

The real win would be empowering the user to choose their own rating style. I don't see this happening because it's much harder to push content at users this way.

Re: How Not to Sort by Average Rating (2009)

#92

Post by ghaff »

SamBam wrote: post #9
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

That's an interesting hypothesis, but I'd want to see more evidence that "a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star." Surely you can't prove this simply by noticing that there are many 1- and 5-star reviews, as there could be many other reasons for that. One obvious one: people who strongly like or strongly dislike a product are more likely to take the time to review. I…

>I personally have never felt the need to review something if I felt "meh" about it.

Or for that matter if I got a simple item and it works as expected. What's an Amazon Basics HDMI cable supposed to do? Does 5 stars mean that it turns HD into 4K through magic? But if it does what an HD cable is supposed to do (and seems well-built) I guess I should give it 5 stars?

Re: How Not to Sort by Average Rating (2009)

#93

Post by ehhnetfliz »

BoiledCabbage wrote: post #63
crazygringo wrote: post #58

Earlier quoted context omitted.

Counterpoint: I almost solely rely on the stars histogram in Yelp (available only on the website, not the app), completely ignoring whatever Yelp's calculated "average" is. If a place has more 5-star ratings than 4-star ratings, it's generally amazing. If it has more 4-star ratings than 5-star ratings, it's generally fine but not something particularly special. Just thumbs up/down would eliminate what is, to me, the…

I used to think the same thing until I realized the most accurate and consistent ratings I use on a regular basis is rotten tomatoes. And they're based on strict thumbs up/ down. It ensures votes hold equal weight and that "extreme polar" voters don't skew things. It also avoids the opposite problem of "everything is neutral" vote unless horrible/incredible. RT also handles high brow and low brow well. You get less v…

In my experience, my favorite movies I find via glowing reviews. Rotten tomato completely obscures this view: if all the reviewers kind of like it, it'll get 100%, whereas polarizing films always suffer. I'll take "kids" over "star wars" any day for a better movie. Why? I'm gonna see star wars because i want to, not because I expect a meaningful aesthetic. But Rotten Tomatoes takes the opposite tact, pushing me towards crowd favorites rather than what i might rate highly.

Really this comes down to how terrible one dimensional comparisons are: it only measure popularity, which is a terrible filter for quality.

Re: How Not to Sort by Average Rating (2009)

#94

Post by mrexroad »

ehhnetfliz wrote: post #91
hyperpape wrote: post #15

Earlier quoted context omitted.

John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…

Ehh, netflix switched to that. It's even less useful now: there's no way to indicate you really like a show vs it's not terrible; this means your taste approximately correlates with abailable content, not content you prefer. The real win would be empowering the user to choose their own rating style. I don't see this happening because it's much harder to push content at users this way.

To be fair, Netflix is less interested in that you liked the thing itself, and more interested in the attributes of the films you liked or watched to the end. Note, this is based on an article from 2014 [1], but a good read nonetheless.

[1] https://www.theatlantic.com/technology/archive/2014/01/how-n...

Re: How Not to Sort by Average Rating (2009)

#95

Post by ghaff »

paulgb wrote: post #73
grandalf wrote: post #55

Earlier quoted context omitted.

Excellent point. Per your question I'm also curious if there is a way to make aggregate ratings more useful as a quality measurement. For instance, if I see a product on Amazon with a 4.8 average rating but notice a lot of very angry 1 star ratings, I'm likely to infer that there may be quality control problems. Amazon displays a histogram so the shopper can assess the meaning of the distribution heuristically. There…

Somewhat related, I think it was Nate Silver who theorized that given enough time all restaurant reviews trend towards four stars. The theory is that if something gets less than four stars it attracts a niche crowd that appreciates it uniquely (and rates highly), while if it gets five stars it attracts a general crowd that doesn't have a particular appreciation (and rates poorly).

And/or restaurants that everyone hates tend not to stay in business. There's also a more general effect that ratings (and reviews) affect the behaviors of people who haven't rated yet. I wouldn't rate many movies that I see below a 3/meh/OK level. That's not so much because I grade inflate but because I actively seek to avoid movies I'd rate 1 or 2 (and, indeed, mostly 3).

Re: How Not to Sort by Average Rating (2009)

#96

Post by ehhnetfliz »

ThrustVectoring wrote: post #50
dntrkv wrote: post #46

Earlier quoted context omitted.

I've been saying this for years, thumbs up/down is the only system that makes sense to me. Foursquare uses it and I've found their scores to be way more useful than Yelp's. The biggest problem with star ratings is that it's so arbitrary. What is the difference between 3 and 3.5? What is a 1 vs a 2? 3/5 is 60%, that's almost failing when you think about it on a grading scale, if I scored something as a 3/5 I would nev…

Yup. You can't assume that what you think a 3/5 means is the same as what someone else thinks a 3/5 means. You can basically assume that for thumbs up/down. And really, the question you want answered is "how likely am I to like this", and thumbs-up % of the overall population is a decent proxy for that for good reason.

What does a thumbs up mean, though? In a netflix context, am i recommending it to others? Trying to train the recommendation for my own taste? Making sure i rewatch it if I don't remember watching it the first time? What do I do if i like a movie but it's objectively terrible? All of the above questions weigh heavily, and the end result is I just avoid binary voting systems (including voting on hn) and it becomes feature bloat with little use.

Strangely, it gets even harder with the thumbs down—there are vanishingly few things i actively wish didn't exist. Why downvote at all?

Re: How Not to Sort by Average Rating (2009)

#97

Post by ehhnetfliz »

mrexroad wrote: post #94
ehhnetfliz wrote: post #91

Earlier quoted context omitted.

Ehh, netflix switched to that. It's even less useful now: there's no way to indicate you really like a show vs it's not terrible; this means your taste approximately correlates with abailable content, not content you prefer. The real win would be empowering the user to choose their own rating style. I don't see this happening because it's much harder to push content at users this way.

To be fair, Netflix is less interested in that you liked the thing itself, and more interested in the attributes of the films you liked or watched to the end. Note, this is based on an article from 2014 [1], but a good read nonetheless. [1] https://www.theatlantic.com/technology/archive/2014/01/how-n...

Right, but they don't ask me what I like about it. Judging by their recommendations they certainly aren't discerning it well.

Netflix is interesting because their content is pretty bad (compared to say IMDB they have very few movies); recommendations are incredibly useful to pretend its library is much larger than it actually is.

I wonder why they didn't go the amazon route of actual reviews + semantic analysis.

Mostly I just hate cross referencing netflix with reviews to figure out if it's worth my time.

Re: How Not to Sort by Average Rating (2009)

#99

Post by logfromblammo »

throwaway613834 wrote: post #80
dntrkv wrote: post #46

Earlier quoted context omitted.

I've been saying this for years, thumbs up/down is the only system that makes sense to me. Foursquare uses it and I've found their scores to be way more useful than Yelp's. The biggest problem with star ratings is that it's so arbitrary. What is the difference between 3 and 3.5? What is a 1 vs a 2? 3/5 is 60%, that's almost failing when you think about it on a grading scale, if I scored something as a 3/5 I would nev…

To be honest, the fact that 60% is a failing grade is a failure of the grading system, not a fact to take for granted. We've basically lost the entire dynamic range of 0-60% for no good reason.

So you're saying that the measurement of student mastery has no noise floor?

Re: How Not to Sort by Average Rating (2009)

#100

Post by logfromblammo »

dota wrote: post #54
stordoff wrote: post #45

Earlier quoted context omitted.

It also doesn't help when reviews aren't made on the merits of the product. Pretty common on, e.g., Steam: > DOTA 2 users then had the brilliant idea to do the dumbest thing any fanbase can do to a game, flood Metacritic with bad user reviews. The slew of zeros since the forgotten Diretide has dropped the game's user score about two points to a 4.5. https://www.forbes.com/sites/insertcoin/2013/11/02/valve-for...

Half-Life fans are currently leaving negative reviews on Dota 2 because Valve won't make HL3. Recent reviews went from "overwhelmingly positive" to "mixed". https://arstechnica.com/gaming/2017/08/steam-reviewers-bomb-...

Valve should probably just farm out HL3 to Obsidian, and continue printing money with Steam.
Post reply on HN