Recommender Systems: We're doing it (all) wrong

My chief tech/gadget metric of recommendation is 'Would I purchase this $expensive_electronic again?' I'm a big shopper at Amazon, but I don't care about the exact breakdown of stars a certain product gets. My usual method is to look at the total number of reviews (as a metric of popularity/community etc), and then to read the 5 star and 1 star reviews (and any that get voted up as most helpful).

I would love to see Amazon go to a binary recommendation system (thumbs up/thumbs down) with a free text review.

Meta quibble...

Is it just me who starts reading articles like this, only to crash into sentences like this one:

"A Likert scale is a unidimensional scale on which the respondent expresses the level of agreement to a statement - typically in a 1 to 5 scale in which 1 is strongly disagree and 5 is strongly disagree."

And think snarky comments like "I should keep reading this article. Do I 1) strongly disagree or 5) strongly disagree?" (Then flick back to the site that linked to it to post that Snarky comment).

There is a lot of work on learning distance functions. A quick Google search will turn up tens, if not hundreds, of papers on this topic (search for learning distance metric or learning distance function).

The prediction context is somewhat different than the analysis context, I think. For predictive recommender systems, the most relevant part of this analysis is the critique of error measures. It may well be that MSE is not an error measure that aligns with the system's actual accuracy goals (e.g. something like perceived quality of the recommendations).

When it comes down to it, the end goal is just to predict whether someone would like something, and/or present them a list of the things you are most certain they'd like. In the analysis context (as with much of HCI), the scales are being used to draw qualitative conclusions about tasks and preferences, so it makes sense to directly attack erroneous modeling and assumptions, because it can lead to wrong conclusions. But for prediction, erroneous modeling only really matters to the extent that it means we're: 1) optimizing the wrong thing; or 2) doing optimization suboptimally.

#1 is important to get right, but #2 is more of a "whatever works" sort of thing, and we even have fairly good automatic methods for deciding. If treating ratings as numerical data empirically leads to good predictions, then it's fine to do; if not, then it's best avoided. Many recent systems avoid even having a human make those kinds of decisions, by throwing in a giant bag of possible ways of slicing the data, and then handing off the decision about which of them to use, and how to weight them, to an ensemble method. Iirc, that's what the winning Netflix-prize entry was like.