Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

41–50 of 158 posts

Re: How Not to Sort by Average Rating (2009)

#41

Post by svachalek »

paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

There's also the hostage-taking effect, most notable in the iOS app store. "This 1-star review will be changed when you give me what I want."

Re: How Not to Sort by Average Rating (2009)

#42

Post by amelius »

> What we want to ask is: Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what? Wilson gives the answer.

Well, you can't answer that question without making assumptions. And these seem to be missing in the article.

Re: How Not to Sort by Average Rating (2009)

#44

Post by jonknee »

hyperpape wrote: post #15
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…

Even Netflix finally moved over to up/down and they were famous for squeezing every drop out of their previous star based reviews [1]. In theory stars work better, but the issue seems to be everyone has a different ranking system.

For example, Uber seems to think anything but a 5/5 is a failure. I know this so I skew to accommodate, but in my personal ranking system I've only had a couple 5 star rides (someone really going above and beyond).

Up/down with an optional qualifier afterwards (e.g. "why were you unhappy?" after a thumbs down) seems to remove a lot of confusion.

[1] https://en.wikipedia.org/wiki/Netflix_Prize

Re: How Not to Sort by Average Rating (2009)

#45

Post by stordoff »

paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

It also doesn't help when reviews aren't made on the merits of the product. Pretty common on, e.g., Steam:

> DOTA 2 users then had the brilliant idea to do the dumbest thing any fanbase can do to a game, flood Metacritic with bad user reviews. The slew of zeros since the forgotten Diretide has dropped the game's user score about two points to a 4.5.

https://www.forbes.com/sites/insertcoin/2013/11/02/valve-for...

Re: How Not to Sort by Average Rating (2009)

#46

Post by dntrkv »

hyperpape wrote: post #15
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…

I've been saying this for years, thumbs up/down is the only system that makes sense to me.

Foursquare uses it and I've found their scores to be way more useful than Yelp's.

The biggest problem with star ratings is that it's so arbitrary. What is the difference between 3 and 3.5? What is a 1 vs a 2? 3/5 is 60%, that's almost failing when you think about it on a grading scale, if I scored something as a 3/5 I would never use that product or service again, yet, many of the best restaurants are rated 3/5 on Yelp.

Unless the user has some scoring system in place for different qualities of the product or service, there is no way you can get anything resembling an accurate score.

I would never trust a user to accurately assess a score given 10 different options (.5-5) but I would be way more likely to trust a user to say either "I like this product" or "I do not like this product."

But yes, the Wirecutter approach works great, but it just doesn't scale.

Re: How Not to Sort by Average Rating (2009)

#47

Post by stordoff »

gknoy wrote: post #28
paulgb wrote: post #17

Earlier quoted context omitted.

That's a good alternative hypothesis. It could also be that people's experience with a product really is bimodal: if I order an alarm clock that works as advertised, it is easy to get 5 stars, if it doesn't work at all, it's 1 star. Your explanation works better for why the distribution persists in books and movies though. In any case, I find the mechanism design angle interesting regardless of the behavioral angle :…

I've often hypothesized that most people are more likely to leave a negative review when they are upset, than a positive review when they like something. It certainly holds in my case: I nearly never review things, because it's a giant hassle, so I have to feel really strongly to be willing to spend the time on a review.

It makes sense - a product doing what it says it will doesn't stand out / incentivise a review, whereas a bad product makes you want to caution others / get some sense of justice on the company (not sure of a better way to word that).

Re: How Not to Sort by Average Rating (2009)

#48

Post by stordoff »

ocfnash wrote: post #38
coldtea wrote: post #31

Earlier quoted context omitted.

Would it be more accurate though?

I don't know; no doubt we could debate the point at length. Nevertheless I stand by my original remark. IMHO the conflation of "Chinese" with "cheap crap" is: * Not relevant * Debatable / Subjective * Divisive and thus distracts attention from the actual point being made.

FWIW, in this specific case, part of the problem can be that items are shipped from China (so longer lead times etc.).

Re: How Not to Sort by Average Rating (2009)

#49

Post by loisaidasam »

Here's a SO post w/ a python implementation:

https://stackoverflow.com/questions/10029588/python-implemen...

The accepted answer uses a hard-coded z-value.

In the event that you want a dynamic z-value like the ruby solution offers, I just submitted the following solution:

https://stackoverflow.com/questions/10029588/python-implemen...

Re: How Not to Sort by Average Rating (2009)

#50

Post by ThrustVectoring »

dntrkv wrote: post #46
hyperpape wrote: post #15

Earlier quoted context omitted.

John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…

I've been saying this for years, thumbs up/down is the only system that makes sense to me. Foursquare uses it and I've found their scores to be way more useful than Yelp's. The biggest problem with star ratings is that it's so arbitrary. What is the difference between 3 and 3.5? What is a 1 vs a 2? 3/5 is 60%, that's almost failing when you think about it on a grading scale, if I scored something as a 3/5 I would nev…

Yup. You can't assume that what you think a 3/5 means is the same as what someone else thinks a 3/5 means. You can basically assume that for thumbs up/down. And really, the question you want answered is "how likely am I to like this", and thumbs-up % of the overall population is a decent proxy for that for good reason.
Post reply on HN