Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not To Sort By Average Rating

evanmiller.org

Post reply on HN
Print view View original

71–80 of 159 posts

Re: How Not To Sort By Average Rating

#71

Post by imajes »

edw519 wrote: post #39
imajes wrote: post #18

Earlier quoted context omitted.

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regula…

There's nothing wrong with the math. I never said there was. In fact, I praised it as an elegant solution. It just requires better educators to explain it, with analogies and metaphor. You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000. Why would you compromise the data/outcome in a hope to simplify the probl…

re the what/further/hint: good point - pre-coffee grumpiness. :) sorry.

Re: How Not To Sort By Average Rating

#72

Post by JumpCrisscross »

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

»The computer decided not to and I have no idea why

This just means you need a better notification system, not a dumber model. I am fairly sure Netflix and Pandora use complicated recommendation algorithms - we don't complain because the complexity is abstracted away behind labels like "dark cerebral thrillers" or "songs with rhythmic gyrations like Pretty Lights".

Re: How Not To Sort By Average Rating

#73

Post by efsavage »

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

Right, the solution here is that if you're going to sort by Bayesian average or Wilson score confidence interval or something besides what we're familiar with, reframe it to avoid my "something's wrong" detector. Hence the fact that people accept "trending" or "popular" but if I sort by "rating" and the results don't make sense because I expected an average, I don't think "I should probably understand this algorithm". I think "this site is broken and dumb".

Re: How Not To Sort By Average Rating

#74

Post by brazzy »

adamio wrote: post #63
edw519 wrote: post #39

Earlier quoted context omitted.

There's nothing wrong with the math. I never said there was. In fact, I praised it as an elegant solution. It just requires better educators to explain it, with analogies and metaphor. You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000. Why would you compromise the data/outcome in a hope to simplify the probl…

Workers who feel powerless by sophisticated solutions need to learn the solutions, and not be intimidated by them. Yes, assembly line workers need simple steps to do a job. But that of course isn't who this is referring to. In your example managing inventory isn't an assembly line, checklist job. You're not replacing 800 workers with the best. In your example, you wouldn't have 800 people managing inventory. You repl…

Sufficiently advanced sophistication is indistinguishable from obfuscation. And obfuscation is a problem that's not ideally solved by hiring smarter people.

I think that's the point edw519 was trying to make.

Re: How Not To Sort By Average Rating

#75

Post by dpritchett »

davidw wrote: post #31
imajes wrote: post #18

Earlier quoted context omitted.

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regula…

> It just requires better educators to explain it, with analogies and metaphor. So... anyone want to take a stab at explaining that equation to those of us who don't really get it?

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom.

[1] http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...

Re: How Not To Sort By Average Rating

#77

Post by dpritchett »

edw519 wrote: post #39
imajes wrote: post #18

Earlier quoted context omitted.

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regula…

There's nothing wrong with the math. I never said there was. In fact, I praised it as an elegant solution. It just requires better educators to explain it, with analogies and metaphor. You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000. Why would you compromise the data/outcome in a hope to simplify the probl…

I'd imagine the right way to handle this would be Reddit style: Implement the naive solutions (i.e. sorted by average score) alongside the top-performing solutions.

Add in a UX that makes it obvious that there's a choice to be made and allows the user to easily evaluate the results of the naive solution alongside the optimized solution, and end users who are themselves evaluated on overall performance metrics will tend to choose the more effective solution whether they understand it or not.

The addition of metrics to allow for side by side evaluation of the simple strategy versus the optimized strategy is key - someone choosing the unintuitive yet superior strategy will likely have to explain themselves to management at one time or another.

Aan interesting parallel is NFL onfield playcalling decisions. There are plenty of instances where coaches make choices that are statistically unsound but they reinforce the image of a tough, responsible coach. Conversely there are coaches who play by the odds and get lambasted when their decision doesn't pan out.

Re: How Not To Sort By Average Rating

#79

Post by jwp »

NathanRice wrote: post #22

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer. Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item.…

What he said! I'd like to add that a key to squeezing more out of NathanRice's post is the phrase "conjugate prior." Another totally natural thing would be to use a Gaussian prior & likelihood, then update the posterior as ratings arrive. This would take advantage of the ordinality of ratings as NR suggests. Bishop's Machine Learning book goes into this sort of stuff in more depth.

Re: How Not To Sort By Average Rating

#80

Post by trebor »

raldi wrote: post #30
edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

The math doesn't make the comments good. The math makes good comments easier to find, and buries comments with inadequate favor or confidence.

Besides, the math made no effort to write the sentences. ;)

Post reply on HN