They'll only have to field those questions if they display the plain averages despite not sorting on them. If we've established that average rating is the wrong metric, why would you even show it to the user/merchant/whoever? Display the same metric that you sort by, and then the merchants will be perfectly happy with higher-rated items showing up higher on the list, and Big Boss will be happy that you've chosen the…
I really don't think that solves the problem; you look at product X, it has 0 reviews, you give it a 5 star, now it shows 1 rating and 3.7 stars?
So you let people rate it with 5 stars, but then show it as 3.7 magic points. Magic points can be your proprietary ranking system.
You don't need to let them rate with stars and also show it as stars.
>What human can understand it? Lets start with Wilson's midpoint, since that's just high school math. def mid(upvotes:Int, downvotes:Int) = { val total = upvotes+downvotes+0.0 val up = upvotes/total val half = 0.5 val a = total/(4+total) val b = 4/(4+total) a * up + b * half } So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes. It should be…
The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably c…
The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it.
I disagree that people won't be able to understand it - I propose they can't be bothered to understand it. Regular people do not give a shit about algorithms - either the computer/software works or it doesn't. Tech geeks regularly forget this simple fact.
Most people have other things they care about and all they want is to press a button and have an answer come out.
If its instantly clear how the answer is generated, they think about it for a very short while, agree and move on with their lives.
If it's not instantly clear how the answer is generated, they assume the computer is right and move on with their lives.
Lots of people seem to be missing the fact that Amazon changed their algorithm years ago to account for the number of review. For example see: http://www.amazon.com/Stand-Mixers-Small-Appliances-Kitchen/...
Huh. You're right. That seems pretty recent, or I didn't notice the switch-over. Thank goodness.
The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably c…
> "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?" 'Because you have fewer votes than your competitor' I don't think it's that hard to understand.
Right. If you look at most-helpful ratios for product reviews on Amazon, you will see this. 500/600 helpful floats far above 10/10 helpful.
The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably c…
the article also misses the human factor. And intent. for amazon, is it better to show the best product first or to help sell a few of the lacking-rating ones to generate more ratings and work out the uncertainty with real world data instead of crazy math?
I don't think it is a good idea to misuse your customers for that kind of testing.
What I don't get is how in 2012 sites like Amazon are still making this mistake. Amazon is a company that, much like Google, spends millions analyzing user behavior and trying to optimize the workflow (checkout, in their case). This has been the number one complaint I have against Amazon for the past 10 years. And they haven't done a think about it?
When I have the whole body of reviews readily available, I like to just do a Bayesian average. Mix in the average number of reviews at the average review score to keep small data sets from skewing results.
I really don't think that solves the problem; you look at product X, it has 0 reviews, you give it a 5 star, now it shows 1 rating and 3.7 stars?
So you let people rate it with 5 stars, but then show it as 3.7 magic points. Magic points can be your proprietary ranking system. You don't need to let them rate with stars and also show it as stars.
If I saw that happen, I would probably assume that the item was so popular that a bunch of other people had rated it concurrently. Unless the number of ratings were displayed simultaneously. There's a bit of a comprehension barrier in letting people know that the ordering score is not the same as the thing they're inputting, and what the relationship between them is.
I don't know how to solve that problem (if I had to, I'd probably just not show the ranking numbers at all- the user already sees the sorting order, what do they need the raw numbers for?), but that's OK, 'cause I prefer to sidestep it- just don't use a graded rating scale like "5 stars". At least not unless you have a really good justification and explanation of what precisely the scale means. 5-star rating systems tend to result in large numbers of 5s and large numbers of 1s anyway, so you might as well just go with a binary like/dislike system. The fact that a ranking is a real number and your input is an "up"/"down" enumeration makes it clear that the ranking you see is not the same thing as the data that you put in.
It could also be useful from a defusing confusion standpoint to make the ranking more complicated by mixing in an age factor (like HN does) or some other extra band of information. Then if a user doesn't immediately understand the relation between their rating and the final ranking, they can easily write it off as being due to some other factor that's not in their control anyway.