How Not To Sort By Average Rating • Hacker News Board

Re: How Not To Sort By Average Rating

#61

Post by IanDrake » Tue, Apr 03, 2012, 3:31 PM UTC

imajes wrote: post #18

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regula…

I thought he was talking about the end user. They may not understand why something with 209 thumbs up and 100 thumbs down ranks better than one with 5 thumbs up (100% positive).

Re: How Not To Sort By Average Rating

#62

Post by padobson » Tue, Apr 03, 2012, 3:31 PM UTC

It seems to me there's still a problem at the point of data collection. Not all +1's are equal.

This algorithm needs to be paired with another algorithm that weights each plus one according to each user's ability to plus one something that gets a lot of plus ones.

I couldn't hope to do the math for something like that, but I'd sure like to talk to someone that could.

Re: How Not To Sort By Average Rating

#63

Post by adamio » Tue, Apr 03, 2012, 3:34 PM UTC

edw519 wrote: post #39

imajes wrote: post #18

Earlier quoted context omitted.

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regula…

There's nothing wrong with the math. I never said there was. In fact, I praised it as an elegant solution. It just requires better educators to explain it, with analogies and metaphor. You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000. Why would you compromise the data/outcome in a hope to simplify the probl…

Workers who feel powerless by sophisticated solutions need to learn the solutions, and not be intimidated by them.

Yes, assembly line workers need simple steps to do a job. But that of course isn't who this is referring to. In your example managing inventory isn't an assembly line, checklist job.

You're not replacing 800 workers with the best. In your example, you wouldn't have 800 people managing inventory. You replace your inventory manager with someone who understands the math (if you want to manage your inventory that way)

Re: How Not To Sort By Average Rating

#64

Post by jules » Tue, Apr 03, 2012, 3:39 PM UTC

_delirium wrote: post #40

NathanRice wrote: post #22

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer. Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item.…

Here's a paper proposing a solution in that space, and which also compares itself to the article linked here (kind of nice to see... papers sometimes fail to cite stuff that's "only" posted online rather than properly published, even if the authors know about it and it's quite relevant): http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... I emailed Miller a while ago to see what he thought of this reply,…

Can anybody replicate their results at Proposition 5? My tests contradict their conclusion, namely that the total score is not monotonic (i.e. when I test I do get that the total score is monotonic).

Re: How Not To Sort By Average Rating

#65

Post by ajross » Tue, Apr 03, 2012, 3:42 PM UTC

It's well explained, informally. The giant equation sitting there without clearly defined parameters is mostly just showing off though. The final "QED" solution that you put at the end of a paper is not the proper form to introduce a concept.

But... so what? Amazon and Urban Dictionary are hardly failing in the market due to their "incorrect" score sorting. The whole problem is a heuristic, it's not amenable to rigorous treatment no matter how many giant equations you club your audience with.

Re: How Not To Sort By Average Rating

#66

Post by NathanRice » Tue, Apr 03, 2012, 3:49 PM UTC

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

Operations research and hard systems modeling are neat. The only gripe I have is that often times the models make simplifying assumptions which hold tenuously at best. One example of this is the Gaussian copula used in VAR. In actuality, neither the marginals nor the copula are Gaussian in the large majority of cases; they ARE stable distributions, but the tails can be absolutely HUGE. The Gaussian distribution is attractive because it is well behaved, and in most cases we don't have enough observations to properly understand the full tail structure of the underlying distribution. As a result, you get quants that make a model and call a small subset of points (which are completely valid) "outliers". Then when we observe the tails of the distribution, things fall apart because over-leveraged bankers made decisions based on the notion that the model accurately reflected reality.

Re: How Not To Sort By Average Rating

#67

Post by bri3d » Tue, Apr 03, 2012, 3:51 PM UTC

raldi wrote: post #30

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

Amusingly the proposed solution in this blog post is almost line-for-line exactly the method Reddit uses to rank comments - proof, I think, that the method isn't "too complicated."

When you're working with a system managing orders, where there's a direct human interaction with the algorithm (like the top-level poster is), simplicity is somewhat important. When you're working with a system that's designed to look like magic to the end user, making the system more magical is often better, because it makes it less intuitive to game.

Re: How Not To Sort By Average Rating

#68

Post by NathanRice » Tue, Apr 03, 2012, 3:59 PM UTC

jules wrote: post #60

NathanRice wrote: post #54

Earlier quoted context omitted.

Absolutely! I just wrote a reply where I alluded to that, unfortunately I didn't refresh and see this post or I would have just plugged you instead. The benefit of the Bayesian treatment here that I want to drill down on is how natural it is to adjust the prior to capture your beliefs about how items should be perceived in the presence of incomplete information. The frequentist approach is fine, but it does not provi…

Well, it depends on how you look at it. The Bayesian MAP approach is basically the same as the frequentist approach with made up data. Instead of using the maximum likelihood estimator p = pos/(pos+neg) you pretend that each new post already has some up/down votes by default, and then you use simply p = pos/(pos+neg). Seems like an even more intuitive explanation of the Bayesian knob to me! Rather than an abstract "a…

Yes, the explanation of "imaginary votes" is by far the simplest way of thinking about the situation. Having mathematical formalism and a rigorously studied methods makes me feel OK about doing it though, otherwise I would be very uncomfortable with the approach :)

I love Bayesian statistics from a conceptual point of view, but the ease with which one ventures into the land of analytic intractability kind of puts me off more complex models. MCMC is such a clumsy tool (in addition to taking forever); variational methods look interesting to me but I don't really feel they are quite there yet.

Re: How Not To Sort By Average Rating

#69

Post by Shorel » Tue, Apr 03, 2012, 4:07 PM UTC

edw519 wrote: post #10

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works. Why I hate it: What human can understand it? I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to a…

Put this code in a class or a library and let a smarter programmer maintain it.

Re: How Not To Sort By Average Rating

#70

Post by mumrah » Tue, Apr 03, 2012, 4:11 PM UTC

There was an article about this a few years back: http://blog.linkibol.com/2010/05/07/how-to-build-a-popularit...

I've found, in practice, a Bayesian weighted average is easy to implement and pretty effective. It's also a good candidate for "stream" processing (i.e., calculating in a single pass)