141–150 of 158 posts
This sort of reminds of "voting theory" and if I recall it was proven by I think a nobel prize winner that there cannot be a fair winner. Obviously it's not entirely analogous but I would not be surprised if it mapped over to this domain. Edit: on mobile so late on the link to Kenneth Arrow https://en.m.wikipedia.org/wiki/Arrow%27s_impossibility_theo...
What about having a scaling factor to adjust the impact of quantity (total) of individual ratings as needed? Rough draft: sort_score = (pos / total) + (W * log(total)) Here, W is the weighting (scaling) factor. Total = positive + negative
score = (pos + a) / (tot + b).
Where aSee this post why that formula follows from Bayesian reasoning: http://julesjacobs.github.io/2015/08/17/bayesian-scoring-of-...
Earlier quoted context omitted.
I envision a panel of emoticons akin to the Facebook reaction set, but where the user can select as many as they want to quickly convey different combinations of their reactions: (thumbs up) I liked this (heart) I loved this (thumbs down) I didn’t like this (smiling face) This made me happy or satisfied (frowning face) This made me sad or disappointed (surprised face) This made me surprised or impressed (angry face)…
Put differently, a set of binary choices: amusing, interesting, sad, ... It's a bit difficult to come up with a good set to rate any thing , but I can see it working for specific topics, like movies or games. Or, one could just let users tag the subject and the interface would display the "weights" of the tags.
A simple utilitarian object? It mostly works or it doesn't.
A movie? Just to start with, there's the rating of the movie itself vs. the rating for this particular DVD. And then there are the dimensions on which the movie itself could be rated.
Or you just throw your hands up in the air and either do a thumbs up/down or a 5 star rating system on the grounds that it's better than nothing.
Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…
John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…
The problem with dichotomous ratings (binary, thumbs up-down) is that they lose a lot of meaningful information without eliminating the problems you're referencing.
That is, the same problems apply to dichotomous ratings, in that people still have tendencies to use the rating scale differently. Some tend to give thumbs up a lot, others down, and people interpret what's good or bad differently. People who are ambivalent split the difference differently.
On top of that, you lose the valid variance in moderate ranges, and actually amplify a lot of these differences in use of the response scale, by forcing dichotomous decisions, because now you've elevated these response style differences to the same level of the "meaningful part" of the response. E.g., maybe one person tends to rate things more negatively than another person, rating 4 and 5 respectively. But when you dichotomize, now that becomes 1 and 2.
The question is whether or not, on balance, the variance associated with irrelevant response scale use is greater than the meaningful variance, and generally speaking studies show the meaningful variance is bigger. In general, you see a small but significant improvement in rating quality going from 2 to 3, and from 3 to 4, and then you get diminishing returns after 4-6 options.
Also, people really don't like being forced to take ambivalence and choose up or down, so in the very least having a middle option is better (unless you want to lose ratings).
It's fairly straightforward to adjust for rating style differences if you have a bunch of ratings of an individual on a bunch of things whose rating properties are fairly well-known. Amazon could do this if they wanted to, and Rotten Tomatoes I think might do something like this already.
RT, in fact, is kind of a bad example, because their situation is so different from typical product ratings, in that you have a small sample of experts who are rating a lot of things. They also are aggregating things that themselves are not standardized-- their use of the tomatometer in part stems from them having to aggregate a wild variety of things, as if everyone on Amazon used a different rating scale, or no rating scale at all. Note too that there's then a "filtering" process involved by RT. Finally I also feel obliged to note they do have ratings and not just the tomatometer, which I've started paying attention to after realizing that things like Citizen Kane show up as having the same tomatometer score as Get Out--a fine movie but not the same.
The game theory angle is interesting to think about. It's something I don't deal with usually because in the situation I'm used to, the raters don't have access to other rater's ratings. That's one solution, but impractical. A sort of meta-rating is one solution--a lot like Amazon's "helpfulness" ratings. It's imperfect but probably does well in adjusting for game theory-type phenomena, like retaliatory rating, etc.
Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…
Individual preferences cannot be aggregated into something that resembles a preference ranking. The most cited formalization of this is Arrow's impossibility theorem, but choice aggregation is this whole theory. _Judgement_ is a slightly different problem. There's an entire issue (#145) of the _Journal of Economic Theory_ on this, but the panorama is still quite bleak, and the reddit approach is far from state-of-the…
That's what's always annoyed me with Amazon's "sort by average rating" setting. I want to see the top 10 or so items by rating to give me a baseline to investigate from, but instead I get page after page of cheap Chinese crap with one 5-star review each from the resident fake reviewer. Worse than useless. Even a simple change like adding a "show only items with a minumum of X reviews" would be a godsend.
I loved it, but I noticed that not too long after (maybe a year?) they removed it. My sense was that small businesses were complaining that the system was unfairly benefiting larger businesses. E.g., if you have a new product, using the lower bound or something similar is unfair because it penalizes you for being new, relative to established players.
Honestly, I can see that perspective too (which is missing from the linked piece), and am not really sure what to do about it. The linked piece comes at it from the perspective of consumer risk minimization, and not from the perspective of the producer, which Amazon also has to contend with.
The solution is probably to allow sorting by both.
Earlier quoted context omitted.
John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…
Ehh, netflix switched to that. It's even less useful now: there's no way to indicate you really like a show vs it's not terrible; this means your taste approximately correlates with abailable content, not content you prefer. The real win would be empowering the user to choose their own rating style. I don't see this happening because it's much harder to push content at users this way.
There's also things to consider, like time, that becomes relevant. Dichotomous ratings are known to be inferior statistically speaking, but they are faster, so there's a convenience angle. Tradeoffs.
These discussions always get frustrating to me because there's so much armchair ad hoc stuff that goes on when there's a huge scientific literature on this already.
People also don't seem to be aware of the assumptions they're making. About ratings being skewed, for example: for a lot of products, people probably do kind of want to know basically "is this meeting my needs?" and then everything is just a decrement away from that. Laundry detergent, for example, is something where I want it to clean my clothes well without damaging them. Why should that be normally distributed?
Also, there's a difference between ratings and how they're used. My guess is that 1-3 star rating variance is meaningful from an experiential point of view, but not from a purchasing point of view. That is, if you had the choice of a 3-star product or a 1-star product, I think people would prefer the 3-star product. When we say "1-3 stars don't matter" we don't actually mean that, we mean that they don't matter because it's below our threshold of what we'd be willing to spend money on.
I've made a simple plot in Excel here: http://i.imgur.com/adjaLQ9.png
The number of up-votes remains the same, while down-votes increases linearly. The scoring declining line in grey is the score.
Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…
I wonder if a system that assigned weights to each individual user's rating based on that user's rating history could help there - if a user always rates products with 5-stars, then another 5-star rating shouldn't have nearly as much weight as one coming from a user that gives a fairly balanced range of ratings. I'm not sure if that would actually work better in practice, but it's at least an interesting idea.
I think the article is missing something visual to demonstrate the actual scoring at work. I've made a simple plot in Excel here: http://i.imgur.com/adjaLQ9.png The number of up-votes remains the same, while down-votes increases linearly. The scoring declining line in grey is the score.
