Live data from Hacker News

Show HN: Improving search ranking with chess Elo scores

zeroentropy.dev

61–67 of 67 posts

Re: Show HN: Improving search ranking with chess Elo scores

#61
post #54
post #11

Earlier quoted context omitted.

We could change the title to "Improving search ranking with chess Elo scores". Anybody object? Edit: ok, done. Submitted title was "Show HN: Improving RAG with chess Elo scores".

They don't use Elo scores. See my comment above, the loss function is adopted from Bradley-Terry.

Bradley-Terry and Elo scores are equivalent mathematical models! The fundamental presumption is the same Thurstone model - that an individual's skill in a particular game is a normally distributed random variable around their fundamental skill.

We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.

Re: Show HN: Improving search ranking with chess Elo scores

#63
post #61
post #54

Earlier quoted context omitted.

They don't use Elo scores. See my comment above, the loss function is adopted from Bradley-Terry.

Bradley-Terry and Elo scores are equivalent mathematical models! The fundamental presumption is the same Thurstone model - that an individual's skill in a particular game is a normally distributed random variable around their fundamental skill. We did experiment with a Bradley-Terry loss function ( https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw ), but we found that even better was to calculate Elo scores, do cross-query bi…

->Bradley-Terry and Elo scores are equivalent mathematical models! No, they are not equivalent mathematical models, they are equalivant in terms of calculation of score function(logistic) given equivalent scale factors. Such that, Bradley-terry: 1/(1 + e^(x(r_B - r_A))) and Elo rating: 1/(1 + 10^((r_B - r_A)/y)), then equivalance requires x = ln(10)/y. More importantly, Elo rating is online scoring system, meaning it takes into accoun the sequence of the events. From your blog post, I understand that you are not updating the scores after after each event. In other words, Elo rating can be interpreted as an incremental fitting of a Bradley-Terry (using similar logistic) model but not the same!

-> The fundamental presumption is the same Thurstone model The Thurstone model is similar, and as you said it assumes normal (as opposed to logistic) using probit link function. It predates both models and due to computational constraints, you can call Bradley-Terry and Elo rating computationally convenient approximation of the Thurstone model.

-> We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw) The math is correct. Thanks for sharing. Indeed, if you do it with incremental updating, you will lose the differentiability given the next winning probability is dependent on the previous updates. Call it what you want, but note that this is not truly and Elo rating which leads misunderstanding. It is Bradley-Terry given you do batch updates which you take extra steps to connect with Elo score, as shown in the link.

Lastly, normal and logistic distribution will lead to log(0) in evaluations which results inf in loss. As I can see from you upper comment, you try add uniform(0.02) as ad-hoc fix. An elegant fix to that is use heavy-tailed distribution such as Cauchy.

Re: Show HN: Improving search ranking with chess Elo scores

#64
post #58

You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).

In our case training and inferencing the models takes days, calculating all of the ELOs take 1min haha. So we didn't need to optimize the calculation. But, we did need to work on numeric stability! I have our calculations here: - https://hackmd.io/@-Gjw1zWMSH6lMPRlziQFEw/B15B4Rsleg tldr; wikipedia iterates on , but that can go to zero or infinity. Iterating on stays between -4 and 4 in all of our observed pairwise ma…

I am working on post-training and evaluation tasks mostly, and I built Evalica as a convenient tool for my own use cases. The computation is fast enough to not bother the user, but the library does not stand in my way during the analysis.

Re: Show HN: Improving search ranking with chess Elo scores

#65
post #25

I have a paper that got denied but it was about using 2AFC sorting to do this instead of elo. It has a defined end unlike elo scores. The code is on my github and focuses on humans sorting images but basically if you have a python sort function, you put your comparison as the key instead of assigning the comparison a numeric score. Then the algorithm does the rest Code: https://github.com/Neywiny/merge-sort Conferenc…

I would have been curious to glance at the paper (poster? whatever it is) but it's paywalled. Is there any particular reason it isn't on arxiv?
Post reply on HN