Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

151–158 of 158 posts

Re: How Not to Sort by Average Rating (2009)

#151

Post by oconnor663 »

folli wrote: post #137
nehushtan wrote: post #8

Earlier quoted context omitted.

What's crazy is everyone knows Amazon's ranking is crap, except apparently Amazon - and it's been crappy in the same way for 10 years.

For Amazon (and equally large companies) I usually tempted to put the proverb "don't attribute to malice which is adequately explained by stupidity" on its head. There's definitely a financial reason behind this.

I'd hasten to add "don't attribute to stupidity what is adequately explained by people having to deal with more than one problem at the same time."

Re: How Not to Sort by Average Rating (2009)

#152

Post by pcollins123 »

paulgb wrote: post #4

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Med…

For every service there are very relevant factors about what makes the product good or bad. If you don't separate these 2 or 3 factors, then over time everything becomes a score of 3.6.

In the case of Amazon, the relevant options are:

   1. Likert scale of quality:
       a junk, just don't buy it
       b cheap and works good enough for occassional use
       c higher quality: willing to spend more and you'll get a much better outcome.
       d overpriced

   2. bad shipping, bad vendor, poor customer service
I hate seeing a bad review for a product based on the last item, they're normally outlier issues or whiners and I normally try to filter them out.

In the case of rotten tomatoes it is, again a different set of parameters.

Re: How Not to Sort by Average Rating (2009)

#154

Post by tabtab »

jules wrote: post #142
tabtab wrote: post #117

What about having a scaling factor to adjust the impact of quantity (total) of individual ratings as needed? Rough draft: sort_score = (pos / total) + (W * log(total)) Here, W is the weighting (scaling) factor. Total = positive + negative

That formula will give an item a higher score the more down votes it gets. A better approach is score = (pos + a) / (tot + b). Where a See this post why that formula follows from Bayesian reasoning: http://julesjacobs.github.io/2015/08/17/bayesian-scoring-of-...

I'm not sure what you mean in the 1st sentence. Example? The problem with the 2 weights is that it's 2 values that have to be given, and for large quantities neither makes much difference. It's why I used log().

Re: How Not to Sort by Average Rating (2009)

#155

Post by antod »

adrianratnapala wrote: post #131
andrewflnr wrote: post #122

Earlier quoted context omitted.

I would actually say it's often not strict enough. In what serious field is it acceptable to only know, say, 70% of the material? Do you want to drive on a bridge designed by an engineer who only got 70% on their exams? It depends on how the test is structured, really, but unless it was one of those tests designed to bring smart people to their knees, I'd rather not.

We probably cross bridges designed by engineers who only got 70% on their exams all the time. That was pretty satisfactory score when I was in Uni.

Yeah - Exam performance from a decade or two ago is quite irrelevant for evaluating senior design engineers.

I wouldn't trust an engineering graduate who scored 100% on all their exams to design a bridge at all. Where as someone with 10+yrs relevant experience but who got 60-70% in their exams would be preferable to me.

Mastery of the math isn't that relevant due to all the design standards you have to understand and comply with anyway, while all the little pragmatic solutions to real world constraints (incl how the builders work and what they need to be effective) learnt from experience and mentoring from your senior peers are far more important.

Re: How Not to Sort by Average Rating (2009)

#156

Post by BoiledCabbage »

stinkytaco wrote: post #72
BoiledCabbage wrote: post #63

Earlier quoted context omitted.

I used to think the same thing until I realized the most accurate and consistent ratings I use on a regular basis is rotten tomatoes. And they're based on strict thumbs up/ down. It ensures votes hold equal weight and that "extreme polar" voters don't skew things. It also avoids the opposite problem of "everything is neutral" vote unless horrible/incredible. RT also handles high brow and low brow well. You get less v…

Rotten Tomatoes is good and predicting a movie I (or others) like, but not really at "ranking". Zootopia, one of their top movies of 2016 and a 98% rating, is a good movie, but one I'm unlikely to pursue again. The Godfather (with a 99%) rating, is a movie I will pick up on Blu Ray and revisit many times. It's far more than 1% better than Zootopia. So RT is good at predicting "should I watch this movie I haven't watc…

> Rotten Tomatoes is good and predicting a movie I (or others) like, but not really at "ranking". Zootopia, one of their top movies of 2016 and a 98% rating, is a good movie, but one I'm unlikely to pursue again

It feels like you're mixing together two different arguments. Rotten Tomatoes is good at predicting whether someone will like a movie. What is "ranking"? That is a very undefined concept. Ranking of what? It's clearly not ranking of likelyhood of a person liking a movie because rotten tomatoes already does that.

Later you mention likelihood of repeat watchings of a movie. Rotten Tomatoes thumbs up or down based on whether someone liked a movie, as a result it produces a metric on likelihood of someone liking of movie. Instead if rotten tomatoes immediately after watching a movie, asking "Did you like this movie?", asked "Would you watch this movie again?" then it would produce an indicator of re-watchability.

Up/down doesn't matter - it's the question that's being asked.

note the caveat RT obviously doesn't actually ask critics these questions, they read and judge their reviews and interpret them as answering those questions.

Re: How Not to Sort by Average Rating (2009)

#157

Post by shaftway »

alexvay wrote: post #148

I think the article is missing something visual to demonstrate the actual scoring at work. I've made a simple plot in Excel here: http://i.imgur.com/adjaLQ9.png The number of up-votes remains the same, while down-votes increases linearly. The scoring declining line in grey is the score.

Here's a 3d graph showing this as a function of upvotes and downvotes. I think it's clearest with

x: [0, 100] y: [0, 100] z: [0, 1]

https://www.google.com/search?q=graph+((x+%2B+1.9208)+%2F+(x...)

Re: How Not to Sort by Average Rating (2009)

#158

Post by jules »

tabtab wrote: post #154
jules wrote: post #142

Earlier quoted context omitted.

That formula will give an item a higher score the more down votes it gets. A better approach is score = (pos + a) / (tot + b). Where a See this post why that formula follows from Bayesian reasoning: http://julesjacobs.github.io/2015/08/17/bayesian-scoring-of-...

I'm not sure what you mean in the 1st sentence. Example? The problem with the 2 weights is that it's 2 values that have to be given, and for large quantities neither makes much difference. It's why I used log().

The log(total) term increases without bound whereas the pos/tot term is at most 1, so in the limit of a lot of votes you will beat an item with fewer votes even if all your votes are downvotes.

That there are two configurable parameters is a good thing. One parameter controls how much of a penalty you get for having few votes, the other controls how many votes count as "few".

Post reply on HN