For movies a histogram may not be helpful (other than highlighting polarizing films), but I find it invaluable for electronics. While I also read reviews, I find that with electronics 1 star reviews (and sometimes 2 star) are usually given for DOA or other serious failure. I can do a quick calculation of defect rate by calculating the percentage of 1 star reviews. If it's 20% (or above norms for that product type) I avoid it. And yes, I realize that the reported defect rate is likely to be higher than the actual defect rate, but it's far better than nothing.
Why Ratings Systems Don't Work
151–160 of 165 posts
Re: Why Ratings Systems Don't Work
#152I'm surprised this article, nor the discussion here doesn't address the main issue here: Just because you use numbers doesn't mean your data is quantitative. Star scores are an attempt to map a qualitative experience (enjoyment of the film) with some quantitative measure. Which is fine if you just want to get a sense of 'how much' somebody liked a something. If I say I give scotch A a 5 scotch B a 3 and scotch C a 4…
I've had this intuition as well, and tried to put it to use when redesigning the Ubuntu Software Center's ratings system. Ratings are fundamentally ordinal data -- higher is better, but the difference between 4 and 5 is not the same as the difference between 3 and 4.
This implies that the arithmetic mean is a broken concept, however the _median_ should still survive intact. I thought about ways to implement this in Software Center, however I'm still not quite sure what a good algorithm for ordinal rating data would look like.
Please feel free to post ideas on this stackexchange question: http://stats.stackexchange.com/questions/19115/how-do-i-sort...
Re: Why Ratings Systems Don't Work
#153The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…
Regarding the unhelpfulness of online reviews, my company has problems with manufacturers/sellers writing 5-star reviews of their own product listings (ASIN's) on Amazon. We've begun (manually) data mining 5-star reviews to identify whether each 5-star-reviewer has any other reviews (or wish list, to indicate the possibility of a real user account), then calculating the % of reviews written by no-history user account…
"but internally at Amazon the Seller Performance team is separate from their Community (user review)"
Indeed, my suspicion is that organizational politics have more to do with the lack of a better rating system than any technical limitation.
The approach I use is to read 3 star ratings first before biasing myself with the more extreme ratings. I also check to see what else the reviewer has rated and if there's nothing there then I immediately dismiss the review.
Re: Why Ratings Systems Don't Work
#154If something has 1000 ratings but is only 3.5 stars whereas something with 1 rating and 5 stars, what does that say about the average? Not much. It just says there's more people to rate something and have an opinion about it. Getting people to rate something is difficult in and of itself and only the people who are on the love/hate spectrum will rate something. I like the histograms as it reveals a little into the ra…
"Trustworthiness" of the reviewer is always difficult, especially with movie reviews, because there's never any accounting for taste. I find it hard to rely on aggregated ratings for that reason. When it came to picking movies to watch, I used to love watching Siskel and Ebert, because I knew their tastes. If only Siskel (whose tastes were more like mine than Ebert's) gave a thumbs up, I knew there was a pretty good…
Human curation is still the state of the art; computed curation is a miserable failure that utterly fails to capture my tastes. It's trivially easy to guess what iTunes or Netflix will recommend to me, which indicates to me that the decision taken is tautological.
The goal of a recommendation system should be to expose me to things I wouldn't be likely to find by myself.
Re: Why Ratings Systems Don't Work
#155Earlier quoted context omitted.
Some bad reviews on Amazon are from shipping snafus. If you're going to get any useful info, you have to read the reviews.
Or worse yet, reviews bitching about the price. Edited to add: "If you're going to get any useful info, you have to read the reviews," is so incredibly true. I'm surprised it's not getting more mentions in these comments. The scatter plot is kind of cool, but I'd so much rather have a histogram and actual reviews to check so I can find out why the product got those ratings.
If you could mouse over each scatter plot point and get a corresponding view that explains its position, that would be cool... for nerds...
Re: Why Ratings Systems Don't Work
#156I definitely go by the 4.5 stars == very good, E.g. when I go to Amazon I don't buy some random product with a 4.5 star review -- I search for a specific product or a specific kind of product and then reject candidates which are lousy. How is that not INCREDIBLY useful? Similarly, who goes to a movie simply based on whether it's good or not. In general, if you create any point rating system people who like a thing wi…
I never actually look at the average star rating. I read a couple 5-star reviews, read more 4-star and 2-star reviews, and decide based on that.
Considering how many Amazon 1-star reviews I find that can be summed up as, "UPS sucked," averages are kinda useless.
Re: Why Ratings Systems Don't Work
#157Earlier quoted context omitted.
> "Who is rating the raters?" Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own . You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it. This is the only good way of going about this method. The trick is,…
Is Netflix really the only site which does this nearly-braindead machine learning approach? Once you realize that people have different tastes and you know someone's preferences that is the obvious solution. Or is the process of crawling through that much statistical data that expensive that it can only be offered to paying subscribers?
The more accurate you want to get, the more computationally expensive. Netflix actually did a contest with a million dollar prize to the team that could come up with the most accurate rating prediction algorithm. In the end, the million dollar algorithm was too expensive to implement, so they never ended up using it.
Re: Why Ratings Systems Don't Work
#158Earlier quoted context omitted.
> "Who is rating the raters?" Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own . You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it. This is the only good way of going about this method. The trick is,…
Pandora does something similar. They only have "like" and "dislike" rating. If you like a song they look at other users who liked that song and try to find more songs/bands from the people with your taste. And the other way around for dislike i guess. It works exceptionally well. You just listen to the stream of incoming songs, you never pick songs yourself. After a good song you click like, after a mediocre song you…
Can the same rating calculation used on Hacker News?
Re: Why Ratings Systems Don't Work
#159This article forgot to take into account one HUGE factor in people who use the rating systems.
ATTENTION SPAN.
Yes. People don't have the attention span to independently analyze 5 different scatter graphs of 5 similar products. Sometimes the scatter plots can be actually more confusing and less informing than something simple like a histogram.
I firmly believe that people's attention spans are more captured by things like star ratings and histograms. If they get past the stage of their interest being captured, THEN they read the reviews to find out more in-depth information and opinions. I think it's a system that works well, as Amazon has shown.
Re: Why Ratings Systems Don't Work
#160Earlier quoted context omitted.
> The problem with any commercial website with ratings (Amazon, Yelp, etc., etc.) is that there is HUGE incentive to game the ratings. While I agree that this is a problem, I think a bigger problem is a simple matter of scale: Amazon is huge, and many people buy things, but they don't split reviews and ratings by what kind of person is rating them. If they wanted to make an improvement, why not show me only ratings a…
Like Netflix
In my experience Netflix ratings are total garbage. IMDB and rottentomato ratings are both way more accurate