Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

31–40 of 158 posts

Re: How Not to Sort by Average Rating (2009)

#31

Post by coldtea »

ocfnash wrote: post #16
kstenerud wrote: post #7

That's what's always annoyed me with Amazon's "sort by average rating" setting. I want to see the top 10 or so items by rating to give me a baseline to investigate from, but instead I get page after page of cheap Chinese crap with one 5-star review each from the resident fake reviewer. Worse than useless. Even a simple change like adding a "show only items with a minumum of X reviews" would be a godsend.

I think this comment would be improved by removal of the word "Chinese".

Would it be more accurate though?

Re: How Not to Sort by Average Rating (2009)

#32

Post by dperfect »

What's the best way to apply the suggested solution to a numeric 5-star rating system (the author mentions Amazon's 5-star system using the wrong approach, yet the solution is specific to a rating system of binary positive/negative ratings)?

I suppose one could arbitrarily assign ratings above a certain threshold to "positive" and those below to "negative", and use the same algorithm, but I imagine there's probably a similar algorithm that works directly on numeric ratings. Anyone know? Or if you must convert the numeric ratings to positive/negative, how does one find the best cutoff value?

Re: How Not to Sort by Average Rating (2009)

#33

Post by quantdev »

eeZah7Ux wrote: post #29

This is computationally very heavy, but, more importantly, for practical purposes you want to have a tunable parameter to balance between sorting by pure rating average and sorting by pure popularity. Often you also want to give a configurable advantage or handicap to new entries.

For a fixed confidence level, it looks computationally light weight: a dozen or so multiplications and divisions plus one square root, which could be approximated if needed. There is no inverse normal needed at run time.

Re: How Not to Sort by Average Rating (2009)

#34

Post by amrrs »

dperfect wrote: post #32

What's the best way to apply the suggested solution to a numeric 5-star rating system (the author mentions Amazon's 5-star system using the wrong approach, yet the solution is specific to a rating system of binary positive/negative ratings)? I suppose one could arbitrarily assign ratings above a certain threshold to "positive" and those below to "negative", and use the same algorithm, but I imagine there's probably a…

What we do with 5-star rating system is completely ignore 2,3,4 stars which in a lot of ways just skew our analysis, hence ending up with a new score similar to Nps (5-star minus 1-star) / total stars

Re: How Not to Sort by Average Rating (2009)

#35

Post by asr »

frgtpsswrdlame wrote: post #6
paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

>A look at rating distributions shows that this is in fact how many people behave. This is really interesting, do you have anything where I could read more about it?

I am having a hard time digging it up, but I remember reading some reporting on the Netflix Prize that said (before Netflix abandoned the star system) that many users rated things only one-star or five-star.

But, counter to the OP's point, I wouldn't assume this is an attempt to move the average; I would guess this is for a number of reasons, including because it's too much mental energy to decide if a product (film) is worth four or five stars, if you rate something you are often just trying to say "liked it" or "didn't like it."

Re: How Not to Sort by Average Rating (2009)

#36

Post by dbaupp »

dperfect wrote: post #32

What's the best way to apply the suggested solution to a numeric 5-star rating system (the author mentions Amazon's 5-star system using the wrong approach, yet the solution is specific to a rating system of binary positive/negative ratings)? I suppose one could arbitrarily assign ratings above a certain threshold to "positive" and those below to "negative", and use the same algorithm, but I imagine there's probably a…

The author has also written http://www.evanmiller.org/ranking-items-with-star-ratings.ht... .

Re: How Not to Sort by Average Rating (2009)

#37

Post by overcast »

amrrs wrote: post #34
dperfect wrote: post #32

What's the best way to apply the suggested solution to a numeric 5-star rating system (the author mentions Amazon's 5-star system using the wrong approach, yet the solution is specific to a rating system of binary positive/negative ratings)? I suppose one could arbitrarily assign ratings above a certain threshold to "positive" and those below to "negative", and use the same algorithm, but I imagine there's probably a…

What we do with 5-star rating system is completely ignore 2,3,4 stars which in a lot of ways just skew our analysis, hence ending up with a new score similar to Nps (5-star minus 1-star) / total stars

Why bother have a 5-star rating system then? Sounds like Netflix went down the right path, with thumbs up or thumbs down. People either zero it out, or give it five stars.

Re: How Not to Sort by Average Rating (2009)

#38

Post by ocfnash »

coldtea wrote: post #31
ocfnash wrote: post #16

Earlier quoted context omitted.

I think this comment would be improved by removal of the word "Chinese".

Would it be more accurate though?

I don't know; no doubt we could debate the point at length.

Nevertheless I stand by my original remark. IMHO the conflation of "Chinese" with "cheap crap" is:

  * Not relevant

  * Debatable / Subjective 

  * Divisive
and thus distracts attention from the actual point being made.

Re: How Not to Sort by Average Rating (2009)

#39

Post by poorman »

I reference this article constantly at Untappd.

When we were building the NextGlass app, I took much of this into consideration for giving wine and beer recommendations.

We recently ran the query on the Untappd database of 500 million checkins and it yielded some interesting results. The "whales" (rare beers) bubbled to the top. I assume this is because users who have to trade and hunt down rare beers are less likely to rate them lower. The movie industry doesn't have to worry about users rating "rare movies", but I would think Amazon might have the same issue with rare products.

Re: How Not to Sort by Average Rating (2009)

#40

Post by gleenn »

thanatropism wrote: post #22

Arguably what Urban Dictionary is doing is to weigh by "net favorability" in some sense and quantity of votes. Quantity of votes correlates to relevance, particularly because UD is meant to represent popular usage.

We actually switched to Wilson score. Doing it later has some weird effects, when you've already have a lot of people typically voting on the first definition, and then suddenly the order gets switched because something has a higher ratio giving it higher confidence. We're honestly not sure it's done anything that great for UD, sometimes simple is just better.
Post reply on HN