How Not to Sort by Average Rating (2009)
evanmiller.org
How Not to Sort by Average Rating (2009)
1–10 of 62 posts
Re: How Not to Sort by Average Rating (2009)
#2Previously: https://news.ycombinator.com/item?id=478632
Re: How Not to Sort by Average Rating (2009)
#3Also discussed in Cameron Davidson-Pilon's Bayesian methods for Hackers in the context of Reddit ups/downs: http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabil...
For Amazon, though, which is the example in Evan Miller's post, I don't really get why you'd first dichotomize the five-star rating into positive vs. negative and then use Wilson intervals. Just construct a run-of-the-mill 95% confidence interval for the mean of a continuous distribution and sort by the (still plausible) worst case scenario a.k.a. the lower bound of that: `mean - 1.96 * SE`, where the standard error is `SE = stddev(scores)/sqrt(n)`.
Because of the central limit theorem, you can do this even if scores are not normally distributed and it'll work out too.
Re: How Not to Sort by Average Rating (2009)
#4Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.
Re: How Not to Sort by Average Rating (2009)
#5I have always wondered what amazon was thinking with that way of sorting.
Perhaps it's a deliberate way to spread purchases out over a span of products instead of just the two top products?
Re: How Not to Sort by Average Rating (2009)
#6I have always wondered what amazon was thinking with that way of sorting. Perhaps it's a deliberate way to spread purchases out over a span of products instead of just the two top products?
I think it's about a product discovery. If we always sort this way, new products don't have a chance.
And I don't think Amazon would sort like this, it would make more sense for them to use hn/reddit way to sort items that give a chance for the new items to get to the top.
Re: How Not to Sort by Average Rating (2009)
#7Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.
I agree. It's a good solution for all cases where the intent is to have a negative vote exactly cancel out a positive.
The method in the article combines a quality rating with a quantity rating, but it's a bit unwieldy and difficult to tune intuitively. It seems to me for a lot of purposes you might get a sufficiently similar effect by using method #1, and then multiplying the result with the sigmoid function applied to the ratio. The advantage of this would be you that the only magical numbers in the formula would be tuning factors you put in yourself.
This seems more appealing to me than "((positive + 1.9208) / (positive + negative) - 1.96 SQRT((positive * negative) / (positive + negative) + 0.9604) / (positive + negative)) / (1 + 3.8416 / (positive + negative))*"
Re: How Not to Sort by Average Rating (2009)
#8Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.
Absolutely agree. The UrbanDictionary ordering seems nearly perfect for most internet purposes.
If this article wants to make its point, it should show cases where its ordering differs from UrbanDictionary.
Re: How Not to Sort by Average Rating (2009)
#9Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.
I agree. It's a good solution for all cases where the intent is to have a negative vote exactly cancel out a positive. The method in the article combines a quality rating with a quantity rating, but it's a bit unwieldy and difficult to tune intuitively. It seems to me for a lot of purposes you might get a sufficiently similar effect by using method #1, and then multiplying the result with the sigmoid function applied…
YMMV, but I'd rather use an elementary and mathematically sound statistical technique than macguyvering something myself. (Though I do understand that, as some people describe in this thread, there can be different purposes to ratings and hence a need for different sorting mechanisms.)
You're right that confidence intervals depend on both quality and quantity, but the reason for this is to account for uncertainty. As n goes up, the standard error goes down to practically 0, and so e.g. a 4.2/5 movie with 100 reviews is still likely to be sorted higher than a 4.1 movie with 200 reviews. Quantity only comes into play when there is very little information to go on, after that quality becomes the driving factor.
Re: How Not to Sort by Average Rating (2009)
#10Earlier quoted context omitted.
I agree. It's a good solution for all cases where the intent is to have a negative vote exactly cancel out a positive. The method in the article combines a quality rating with a quantity rating, but it's a bit unwieldy and difficult to tune intuitively. It seems to me for a lot of purposes you might get a sufficiently similar effect by using method #1, and then multiplying the result with the sigmoid function applied…
YMMV, but I'd rather use an elementary and mathematically sound statistical technique than macguyvering something myself. (Though I do understand that, as some people describe in this thread, there can be different purposes to ratings and hence a need for different sorting mechanisms.) You're right that confidence intervals depend on both quality and quantity, but the reason for this is to account for uncertainty. As…
It's not about macguyvering something, it's about reflecting on how well different solutions stack up with what you actually want to do. There is no one-size-fits-all ranking formula, at least in my opinion.