Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not to Sort by Average Rating (2009)

evanmiller.org

Post reply on HN
Print view View original

131–140 of 158 posts

Re: How Not to Sort by Average Rating (2009)

#131

Post by adrianratnapala »

andrewflnr wrote: post #122
throwaway613834 wrote: post #80

Earlier quoted context omitted.

To be honest, the fact that 60% is a failing grade is a failure of the grading system, not a fact to take for granted. We've basically lost the entire dynamic range of 0-60% for no good reason.

I would actually say it's often not strict enough. In what serious field is it acceptable to only know, say, 70% of the material? Do you want to drive on a bridge designed by an engineer who only got 70% on their exams? It depends on how the test is structured, really, but unless it was one of those tests designed to bring smart people to their knees, I'd rather not.

We probably cross bridges designed by engineers who only got 70% on their exams all the time. That was pretty satisfactory score when I was in Uni.

Re: How Not to Sort by Average Rating (2009)

#133

Post by yoz-y »

ghaff wrote: post #104
yoz-y wrote: post #51

Earlier quoted context omitted.

Maybe the problem is with stars and wording. Currently most of systems are worded (e.g. amazon) in a way that 3 stars is the base and people would add stars if their expectations were exceeded and remove them if they were not met. However at all places that I have seen it is like you say 5 stars is for expectations being met and it only goes downhill from that. I think that a wording and iconography in 4 steps could…

I'm not sure an expectations-based rating system is the norm though. To use an example I gave elsewhere. I order a cable from Amazon. It works. Therefore it met my expectations of a working cable. Yet, I think most people would interpret a 3-star rating as my being lukewarm on my purchase. I'm not. But what the heck do I expect a cable to do other than being a fair price and to work? With respect to movies. Some movi…

For a single data point they are useless. But I thing a rating in aggregate could be informative. Of course, the rating is self defeating as too many people who were positively surprised will raise the expectations.

For movies I would love to have reviews in style: Somebody writes up their expectations before going to the movie, and then rates the movie according to that. I find most "press and critics" reviews useless as if somebody is not a fan of story-less action movies, then why the hell is he reviewing them.

Re: How Not to Sort by Average Rating (2009)

#134

Post by kalleboo »

stinkytaco wrote: post #70
jonknee wrote: post #44

Earlier quoted context omitted.

Even Netflix finally moved over to up/down and they were famous for squeezing every drop out of their previous star based reviews [1]. In theory stars work better, but the issue seems to be everyone has a different ranking system. For example, Uber seems to think anything but a 5/5 is a failure. I know this so I skew to accommodate, but in my personal ranking system I've only had a couple 5 star rides (someone really…

Everyone does have a different ranking system, but Netflix was good at predicting what I would give a movie, so my ranking system was the only one that mattered. Their special sauce in the background made it pretty accurate. The problem with an up/down for me is that it doesn't capture an ambivalent reaction. That means transactions will tend toward the mode, imo. You will do enough to get a thumbs up, that's all.

I often wish it was an up/down/love

Re: How Not to Sort by Average Rating (2009)

#135

Post by Houshalter »

dntrkv wrote: post #46
hyperpape wrote: post #15

Earlier quoted context omitted.

John Gruber has been arguing that the only meaningful way to do ratings is a simple thumbs up/thumbs down. I don't necessarily agree, but I see the appeal. I usually don't want ratings, I want the Wirecutter treatment. Sometimes, I know/care enough to really research the topic, in which case star reviews are relatively unhelpful. The rest of the time, I just want someone trustworthy to say "buy this if you want to pa…

I've been saying this for years, thumbs up/down is the only system that makes sense to me. Foursquare uses it and I've found their scores to be way more useful than Yelp's. The biggest problem with star ratings is that it's so arbitrary. What is the difference between 3 and 3.5? What is a 1 vs a 2? 3/5 is 60%, that's almost failing when you think about it on a grading scale, if I scored something as a 3/5 I would nev…

Just normalize ratings. If the average rating is in the 50th percentile of all ratings on the site, convert the rating to 50%. That way it carries the maximum possible information. If someone rates something 60% that just means it's better than 60% of similar products.

School grading systems serve a completely different purpose and are a terrible comparison.

Re: How Not to Sort by Average Rating (2009)

#136

Post by Houshalter »

icebraining wrote: post #74
stinkytaco wrote: post #72

Earlier quoted context omitted.

Rotten Tomatoes is good and predicting a movie I (or others) like, but not really at "ranking". Zootopia, one of their top movies of 2016 and a 98% rating, is a good movie, but one I'm unlikely to pursue again. The Godfather (with a 99%) rating, is a movie I will pick up on Blu Ray and revisit many times. It's far more than 1% better than Zootopia. So RT is good at predicting "should I watch this movie I haven't watc…

Rotten Tomatoes actually has both ratings, meaning they recognize the limitation you're referring to. In the other, Zootopia has 8.1/10 and The Godfather has 9.2/10, showing that difference in quality.

Also you just aren't the demographic for zootopia. If you have kids then it probably is worth buying and they will watch it many times. There are so many genres of films, it's best to compare within a single genre and not between.

Re: How Not to Sort by Average Rating (2009)

#137

Post by folli »

nehushtan wrote: post #8
kstenerud wrote: post #7

That's what's always annoyed me with Amazon's "sort by average rating" setting. I want to see the top 10 or so items by rating to give me a baseline to investigate from, but instead I get page after page of cheap Chinese crap with one 5-star review each from the resident fake reviewer. Worse than useless. Even a simple change like adding a "show only items with a minumum of X reviews" would be a godsend.

What's crazy is everyone knows Amazon's ranking is crap, except apparently Amazon - and it's been crappy in the same way for 10 years.

For Amazon (and equally large companies) I usually tempted to put the proverb "don't attribute to malice which is adequately explained by stupidity" on its head. There's definitely a financial reason behind this.

Re: How Not to Sort by Average Rating (2009)

#138

Post by enobrev »

yoz-y wrote: post #51
jonknee wrote: post #44

Earlier quoted context omitted.

Even Netflix finally moved over to up/down and they were famous for squeezing every drop out of their previous star based reviews [1]. In theory stars work better, but the issue seems to be everyone has a different ranking system. For example, Uber seems to think anything but a 5/5 is a failure. I know this so I skew to accommodate, but in my personal ranking system I've only had a couple 5 star rides (someone really…

Maybe the problem is with stars and wording. Currently most of systems are worded (e.g. amazon) in a way that 3 stars is the base and people would add stars if their expectations were exceeded and remove them if they were not met. However at all places that I have seen it is like you say 5 stars is for expectations being met and it only goes downhill from that. I think that a wording and iconography in 4 steps could…

This is how I see it as well. It would also likely get rid of comments / ratings like "I would give this 0 stars if I could". I think the numbering you mention is just as essential (especially 0).

The way I generally rate things falls in line with this idea:

5: Excellent

4: Pretty Good

3: Average; Unsurprising. Not Impressed, Not Disappointed

2: Kinda Sucks

1: Run Away

This definitely doesn't seem to be how everyone else is using the 5-star scale.

Re: How Not to Sort by Average Rating (2009)

#139

Post by fnordian_slip »

bllguo wrote: post #103
BearGoesChirp wrote: post #56

Earlier quoted context omitted.

It would be a lot more work, but one could check the validity of ratings based on how other users rate things compared to this user. Say there are 4 games. Most users who rate all 4 rate them similar, except for the 4th game that always gets really low. So 5,5,5,1 is a normal expected rating, but 1,1,1,5 isn't. So 5,5,4,4 from a high rater or 2,2,2,2 from a low rater would be given more weight than a 1,1,1,5. Other t…

unfortunately sample sizes will probably decrease drastically when you're looking at the subset of people who rated 4 specific games, or other such cases

Yes, that alone is probably enough to make any system like that come with a huge false-positive rate.

I myself only rarely post a review, and I could easily see how - in a vacuum - my star ratings might look like those of a bot.

Re: How Not to Sort by Average Rating (2009)

#140

Post by hanbura »

waterhouse wrote: post #126
logfromblammo wrote: post #99

Earlier quoted context omitted.

So you're saying that the measurement of student mastery has no noise floor?

It depends on how things are graded. On a multiple-choice test with four choices per question, someone with no knowledge who guesses randomly will get ~25%. On a true-false test, someone with no knowledge gets ~50%. On a project graded by a human, or a worksheet whose answers are real numbers, someone with no knowledge and a hard-eyed grader might well get 0%. Different classes will have different proportions of thes…

>On a multiple-choice test with four choices per question, someone with no knowledge who guesses randomly will get ~25%.

That would be terrible test design. At my (German) university, most Multiple Choice tests give one point for a correct answer, minus half a point for a wrong answer. That way you expect negative points from people who think they know everything but are no better than random guessing, zero points from somebody who knows nothing, some points from someone who can always narrow it down to two choices.

I guess my point is that you can arbitrarily raise the floor with a bad grading scheme, but there's no inherent reason to do that.

Post reply on HN