tl;dr (my interpretation anyway). In utility theory a completely specified preference ordering is the starting point and a utility function can be derived to represent it. These functions are unique only up to a monotone transformation. In the recommender systems we take the quantity as a given and infer the preference order for missing items. If you reassign assign all the 5 stars items to 10 stars, it is perfectly consistent with the ordering, but your inference method may be (will be!) sensitive to such reassignments. To me this suggests you should also be estimating the shape of the best transformation of the rating system. Think of it as an opportunity to reduce bias. The next step would be to see if your particular decision problem is sensitive to such parameters. If not, chuck those parameters and go about your business.
Recommender Systems: We're doing it (all) wrong
21–30 of 38 posts
Re: Recommender Systems: We're doing it (all) wrong
#22A good way to deal with recommendation systems that avoids these problems is via flows on graphs. Here’s one method of converting individual ratings to global rankings. Each alternative A, B, C, etc is a node of the graph. Now, when the user/voter gives 5 stars to A and 4 stars to B, this is interpreted simply as a preference of A over B. This preference contributes a single point to the total flow from B to A. At th…
This also eliminates the inconsistencies between Likert "intervals" across different subjects. Very smooth.
Re: Recommender Systems: We're doing it (all) wrong
#23Put up or shut up (I don't mean this in a mean way... read on). There is a great data set to test this theory... Netflix. This article shouldn't end by just solicitng opinions, but with his results in the Netflix data set.
Hi @kenjackson, this is Xavier here. How do you propose to validate this on the Netflix dataset? It is clear that you cannot use RMSE to compare to other existing approaches, right? The way to go would be to propose a different success measure (i.e. ranking based) and measure how different algorithm perform. And then validate this on users to prove that optimizing RMSE is not as useful. If you give me a few months, I…
If you give me a few months, I might get there. But this is the reason I wrote a blog post and not a paper ;-)
Fair enough. I'll keep my eyes peeled for a paper in a few months. :-)
Re: Recommender Systems: We're doing it (all) wrong
#24Really great point. Slightly off topic, the standard for rating systems seems to be 5 stars, but I prefer 4 star systems because they force you to make a +/- choice with no cop out ambivalence choice. I'd be curious to see the same distance work applied to a four star system.
Why is it a cop-out to be ambivalent?
> Why is it a cop-out to be ambivalent?
Well, I would say it kind of is, but it kind of isn't.
Re: Recommender Systems: We're doing it (all) wrong
#25Really great point. Slightly off topic, the standard for rating systems seems to be 5 stars, but I prefer 4 star systems because they force you to make a +/- choice with no cop out ambivalence choice. I'd be curious to see the same distance work applied to a four star system.
So you are forcing people who are neither in favour nor against something to produce false pro/contra votes. There are some cases where this strategy makes sense like questions about controverse topics. Other than that it produces noise. In my experience, the debate 4 vs 5 options often has more to do with the authors' personality and training than anything else. Surprisingly little is based on solid, non-ambiguous e…
It's not forcing -- most rating systems don't force you to rate -- rating is for when wanting to express your opinion.
And I can see a difference between someone that hasn't voted, and someone that voted a 3-stars in a 5-stars system.
Re: Recommender Systems: We're doing it (all) wrong
#26Treating ordinal ratings (1-5) as nominal labels is not the best solution, IMO. It's true that the "distance" between each rating is not equal, however disregarding the fact that 4 stars is greater than 3 looses quite a bit of useful information.
Re: Recommender Systems: We're doing it (all) wrong
#27tl;dr (my interpretation anyway). In utility theory a completely specified preference ordering is the starting point and a utility function can be derived to represent it. These functions are unique only up to a monotone transformation. In the recommender systems we take the quantity as a given and infer the preference order for missing items. If you reassign assign all the 5 stars items to 10 stars, it is perfectly…
I think 11 stars is a better scale.
Additionally, a great body of work in behavioral psych tells us that humans have a tough time measuring preferences on any absolute scale; however, we can consistently compare two items as better or worse (particularly when they're of the same type, instead of apples versus oranges). "Riffle independence" is a recent method for modeling these kinds of preference distributions, and has been used quite successfully for social curation of the blogosphere - i.e., showing the best set of blogs that span the topic space and have little redundancy.
Re: Recommender Systems: We're doing it (all) wrong
#28Earlier quoted context omitted.
Hi @kenjackson, this is Xavier here. How do you propose to validate this on the Netflix dataset? It is clear that you cannot use RMSE to compare to other existing approaches, right? The way to go would be to propose a different success measure (i.e. ranking based) and measure how different algorithm perform. And then validate this on users to prove that optimizing RMSE is not as useful. If you give me a few months, I…
I just wanted to point out that the link on your blog to PureSVD points to the wikipedia page for Discounted Cumulative Gain.
OOps! Thanks... just fixed it.
Re: Recommender Systems: We're doing it (all) wrong
#29Really great point. Slightly off topic, the standard for rating systems seems to be 5 stars, but I prefer 4 star systems because they force you to make a +/- choice with no cop out ambivalence choice. I'd be curious to see the same distance work applied to a four star system.
Using 4 points instead of 5 merely removes some of the inherent ambiguity, the results are still likely to organize around similar concepts.
For many things, most people don't bother to think about the difference between crap unless you force them to, so in a 4-star system 2 stars becomes the "mediocre" rating while 3 and 4 differentiates between the good ones. How many people really care about a grade difference between D and an F? Likewise, do people really spend that much time making sure their 1 and 2 star ratings form a consistent philosophy of relative crappiness?
If "forcing" people to make a choice about something that is supposed to be a subjective categorization to begin with is probably not helping anything. If you want to force a like/dislike you should get binary data and be done with it.
"Would you eat this? Yes/No."
That's easy to answer accurately. Everyone will agree on what you mean. If you haven't answered that means you don't have an opinion. Beyond that semantic ambiguity is impossible to avoid and gets worse the more numbers you add.
Re: Recommender Systems: We're doing it (all) wrong
#30This article is a great example of why 'computer science' should not assume that they understand statistics or even much in data manipulation.
Then the article is an example of why computer people should be careful on where they learn their statistics!
The article is awash in hand wringing about "interval scale" and "ordinal scale" data without being at all clear on just why someone should care, and for all the rest of the article they should not care.
So, the article has:
"For ordinal data, one should use non-parametric statistical tests which do not assume a normal distribution of the data."
Mostly nonsense. In statistical testing, the normal distribution arises mostly just via the central limit theorem which has quite meager assumptions trivially satisfied by "Likert" scale data.
Then there is:
"Furthermore, because of this it makes no sense to report means of likert scale data--you should report the mode."
Nonsense: The law of large numbers has especially meager assumptions also trivially satisfied by Likert scale data. If you want to estimate expectation, then definitely use the mean and not the mode.
Beyond the law of large numbers, there is also the classic
Paul R. Halmos, "The Theory of Unbiased Estimation", 'Annals of Mathematical Statistics', Volume 17, Number 1, pages 34-43, 1946.
that makes clear that the mean is the most accurate way to estimate expectation.
If you want to use the mode for something, then say what the heck you want to use it for and then justify using the mode as the estimator.
There is:
"In order to defend that ratings can be treated as interval, we should have some validation that the distance between different ratings is approximately equal."
Nonsense. Instead, you get a 'rating', say, an integer in [1, 5]. Now you have it. Use it. For
"validation that the distance between different ratings is approximately equal."
why bother? Besides, "the distance" is undefined here!
For the
"This is a clear indication that users perceive that the distance between a 2 and a 3 is much lower than between a 4 and a 5."
the writer is just fishing in muddy waters.
There is
"All the neighbor based methods in collaborative filtering are based on the use of some sort of distance measure. The most commonly used are Cosine distance and Pearson Correlation. However, both these distances assume a linear interval scale in their computations!"
Nonsense. Just write out the definitions of expectation, variance, covariance, and Pearson correlation and see that sufficient is that the expectation of the squared random variables be finite. There is nothing about "interval scale" in the assumptions.
But why calculate Pearson correlation? When dig into that, again, basically just want some MSE (mean square error) convergence, which again makes no assumptions about "interval scale" data.
There is
"This is my favorite one... The most commonly accepted measure of success for recommender systems is the Root Mean Squared Error (RMSE). But wait, this measure is explicitly assuming that ratings are also interval data!"
Nonsense. There is no such assumption about MSE. The main point about MSE is just that any sequence of random variables (e.g., estimates) that converges in MSE will have a subsequence that converges almost surely. In practice, convergence in MSE is convergence almost surely, and that's the best convergence there can be. So, if your estimates are good in MSE, then essentially always in practice they are close in every sense. Nowhere in this argument is an assumption about "interval data".
This article sounds like 'statistics' from some psycho researcher who has an obsession about interval scales and a phobia about using ordinal scale data! In particular he has high anxieties about being charged with heresy by the Statistical Religious Police! The guy needs 'special help'!
Did I mention that the article is nonsense?