Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recommender Systems: We're doing it (all) wrong

technocalifornia.blogspot.com

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Recommender Systems: We're doing it (all) wrong

#11

Post by logophobia »

I wonder if it's possible to assign a weight to each of the possible scores to correct for the "perceived" distance so you can still use all these existing tools in a statistical valid way.

Also, this observation could be interpreted a bit differently:

> The probability that a user changes her rating between 2 and 3 is almost 0.35 while the probability she changes between 4 and 5 goes down to almost 0.1. This is a clear indication that users perceive that the distance between a 2 and a 3 is much lower than between a 4 and a 5.

It seems a bit counter intuitive that the distance between 3 (neutral) and 4 (positive) is smaller then 4 and 5 (very positive). You could also interpret this differently. When a user changes his mind, he has to change his mind in such a way that the difference is significant enough to also change the review (is the review now a little bit wrong or very wrong). This means that he might actually see the difference between 3 and 4 as larger then 4 and 5, large enough for him/her to change the review. This effect is dampened the amount of time the user actually changes his mind this way. If you look at it in that way then the amount of pairwise inconsistencies are the wrong way to measure the distance between these ordinal categories in this particular case, because there actually might be two mechanisms that cancel each other out.

Re: Recommender Systems: We're doing it (all) wrong

#13

Post by dhimes »

logophobia wrote: post #11

I wonder if it's possible to assign a weight to each of the possible scores to correct for the "perceived" distance so you can still use all these existing tools in a statistical valid way. Also, this observation could be interpreted a bit differently: > The probability that a user changes her rating between 2 and 3 is almost 0.35 while the probability she changes between 4 and 5 goes down to almost 0.1. This is a cl…

Do we know if there is a difference in perception by the respondent if the scale is numerical? A scale of 1-5 might be seen as having different intervals than a scale whose range is strongly-disagree -- strongly agree.

Re: Recommender Systems: We're doing it (all) wrong

#14

Post by xamat »

kenjackson wrote: post #12

Put up or shut up (I don't mean this in a mean way... read on). There is a great data set to test this theory... Netflix. This article shouldn't end by just solicitng opinions, but with his results in the Netflix data set.

Hi @kenjackson, this is Xavier here. How do you propose to validate this on the Netflix dataset? It is clear that you cannot use RMSE to compare to other existing approaches, right? The way to go would be to propose a different success measure (i.e. ranking based) and measure how different algorithm perform. And then validate this on users to prove that optimizing RMSE is not as useful.

If you give me a few months, I might get there. But this is the reason I wrote a blog post and not a paper ;-)

Re: Recommender Systems: We're doing it (all) wrong

#15

Post by kvh »

You could just map 5 to something farther away, like 6. In fact, this is how most ordinal inference techniques work anyways: by taking an interval method and learning cutoffs for your ordered categories. Learning more parameters comes with a big cost though, which is why in practice the cutoffs are often fixed from the get-go. Obviously, ordinal methods have been tried in the literature. There is a reason they are not used in practice though, and that's because the trade-off (being harder to learn vs modelling the data more accurately) is not favorable.

Re: Recommender Systems: We're doing it (all) wrong

#16

Post by pixcavator »

A good way to deal with recommendation systems that avoids these problems is via flows on graphs. Here’s one method of converting individual ratings to global rankings. Each alternative A, B, C, etc is a node of the graph. Now, when the user/voter gives 5 stars to A and 4 stars to B, this is interpreted simply as a preference of A over B. This preference contributes a single point to the total flow from B to A. At this point, we remove all cycles from this flow (there is a standard way to do that) and produce a gradient flow F. The potential function h of this flow, grad h = F, is the rankings.

Re: Recommender Systems: We're doing it (all) wrong

#17

Post by dhimes »

2mur wrote: post #2

My chief tech/gadget metric of recommendation is 'Would I purchase this $expensive_electronic again?' I'm a big shopper at Amazon, but I don't care about the exact breakdown of stars a certain product gets. My usual method is to look at the total number of reviews (as a metric of popularity/community etc), and then to read the 5 star and 1 star reviews (and any that get voted up as most helpful). I would love to see…

Sometimes, learning about the nuances is helpful, and they seem to be in the 4-star (and even 3-star) reviews, especially when there are many 5-star reviews. That is, the reviewer says things like, "I would have given it 5 stars except ___." I zero in on those reviews to try to understand any edge cases in usability or suitability.

EDITED for embarrassing grammar mistakes

Re: Recommender Systems: We're doing it (all) wrong

#18

Post by luvcraft »

Interesting. I've noticed on reccr ( http://reccr.com ), my own recommendation project, that the recommendations for 4 and 5 star ratings are much more accurate than those for 2 and 3 star ratings. I had initially thought that it was simply that there were more 4 and 5 star ratings in the system, and thus more data to base recommendations on, but the "larger perceived gap" between 2 and 3 versus 4 and 5 makes a lot of sense and is probably also a major contributing factor.

Re: Recommender Systems: We're doing it (all) wrong

#19

Post by frak_your_couch »

xamat wrote: post #14
kenjackson wrote: post #12

Put up or shut up (I don't mean this in a mean way... read on). There is a great data set to test this theory... Netflix. This article shouldn't end by just solicitng opinions, but with his results in the Netflix data set.

Hi @kenjackson, this is Xavier here. How do you propose to validate this on the Netflix dataset? It is clear that you cannot use RMSE to compare to other existing approaches, right? The way to go would be to propose a different success measure (i.e. ranking based) and measure how different algorithm perform. And then validate this on users to prove that optimizing RMSE is not as useful. If you give me a few months, I…

I just wanted to point out that the link on your blog to PureSVD points to the wikipedia page for Discounted Cumulative Gain.

Re: Recommender Systems: We're doing it (all) wrong

#20

Post by stewbrew »

macrael wrote: post #3

Really great point. Slightly off topic, the standard for rating systems seems to be 5 stars, but I prefer 4 star systems because they force you to make a +/- choice with no cop out ambivalence choice. I'd be curious to see the same distance work applied to a four star system.

So you are forcing people who are neither in favour nor against something to produce false pro/contra votes. There are some cases where this strategy makes sense like questions about controverse topics. Other than that it produces noise. In my experience, the debate 4 vs 5 options often has more to do with the authors' personality and training than anything else. Surprisingly little is based on solid, non-ambiguous evidence.
Post reply on HN