Little reminder that Elo is a guy, not an acronym :)
Came to comment this. As a consequence, writing it in capital letters "ELO" is wrong.
Show HN: Improving search ranking with chess Elo scores
51–60 of 67 posts
Re: Show HN: Improving search ranking with chess Elo scores
#52> Fit an ELO-style rating system (Bradley-Terry) to turn pairwise comparisons into absolute per-document scores.
There are some conceptual gaps and this sentence is misleading in general. First, this sentence implies that Bradley-Terry is a some sort of an Elo variant, which is not true. Elo rating introduced nearly 10 years later in completely different domain.
They are two completely different ranking systems. Bradley-Terry use ratio-based, while Elo use logistic score function. Scales of the scores are completely different as well as the their sensitivity to the score differences.
Possibly, Bradley-Terry is preffered by the authors due to simpler likelihood evaluation and update doesn't depend on the order of pairwise evaluations.
There is also variants of Elo-rating that use MLE (optimized Elo) and even recently Bayesian Elo. For post-hoc time invariant scores, there is randomized Elo rating and so on.
People like Elo ratings because they are simple to understand. Most of the time, they forget why they developed specifically for chess tournaments. All variants above and dozens more try to improve (fix) one aspect of the Elo ratings, because their application has no 100% clear determination of winner, the update scale parameter is too small or large, matches are played simultaneously, different matches played and so on.
Also, let say one document is always preffered one all LLMs then it has only wins, then MLE will result in flat marginal likelihood for that where the update parameter (c) will inf.
Re: Show HN: Improving search ranking with chess Elo scores
#53Out of curiosity, is there a reason why you are using ELO proper, rather than one of the ELO variants that doesn't make assumptions about the distribution of results? E.g.: https://github.com/pfmonville/whole_history_rating
Hey! We actually did a lot of research into ELO consistency, i.e. to check whether or not the NxN pairwise matrix followed the ELO model. It was a long road that's probably grounds for an entirely separate blog post, but the TLDR is that we observe that: For each document, there is a secret hidden score "s" which is the "fundamental relevance according to the LLM". Then, when we sample (q, d1, d2) from the LLM, the L…
More confused,
1) 0.02 * random.random() != N(0, 0.02)
2) The LLM will sample a normal distribution, this only depends on your c parameter, the absolute scale doesn't matter neither in Bradley-Terry nor in Elo. So saying +-4 and claiming LLM reasoning in Standard normal is ridiculous.
3) > then we get a pairwise matrix with virtually identical statistical properties to the observed pairwise matrices. >>> then did you asked yourselves if I have "statistically identical" pair-wise matrix and observed pairwise matrix, the. why you even bother myself? You can simply use observed pairwise matrix...
Re: Show HN: Improving search ranking with chess Elo scores
#54I would have titled it "Improving ranking..." I like that it works with `sentence_transformers`
We could change the title to "Improving search ranking with chess Elo scores". Anybody object? Edit: ok, done. Submitted title was "Show HN: Improving RAG with chess Elo scores".
They don't use Elo scores. See my comment above, the loss function is adopted from Bradley-Terry.
Re: Show HN: Improving search ranking with chess Elo scores
#55Explanation of Bradley-Terry here: https://stats.stackexchange.com/a/131270/60526
It's such a great and simple algorithm. I feel like it deserves to be more widely known.
I used it at Dyson to evaluate really subjective things like how straight a tress of hair is - pretty much impossible to say if you just look at a photo, but you can ask a bunch of people to compare two photos and say which looks straighter, then you can get an objective ranking.
Re: Show HN: Improving search ranking with chess Elo scores
#56I think Elo style rankings would be good for rating e.g. Uber rides and restaurant reviews. Instead of asking to rate out of 5 stars or similar, where everyone basically ends up giving 5 stars, just ask was it better or worse than your last experience.
Re: Show HN: Improving search ranking with chess Elo scores
#57I think Elo style rankings would be good for rating e.g. Uber rides and restaurant reviews. Instead of asking to rate out of 5 stars or similar, where everyone basically ends up giving 5 stars, just ask was it better or worse than your last experience.
Is this really viable for something like Uber, where most rides aren't really meaningfully better or worse?
Re: Show HN: Improving search ranking with chess Elo scores
#58You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).
In our case training and inferencing the models takes days, calculating all of the ELOs take 1min haha. So we didn't need to optimize the calculation.
But, we did need to work on numeric stability!
I have our calculations here: - https://hackmd.io/@-Gjw1zWMSH6lMPRlziQFEw/B15B4Rsleg
tldr; wikipedia iterates on , but that can go to zero or infinity. Iterating on stays between -4 and 4 in all of our observed pairwise matrices, so it's very well-bounded.
Re: Show HN: Improving search ranking with chess Elo scores
#59Explanation of Bradley-Terry here: https://stats.stackexchange.com/a/131270/60526 It's such a great and simple algorithm. I feel like it deserves to be more widely known. I used it at Dyson to evaluate really subjective things like how straight a tress of hair is - pretty much impossible to say if you just look at a photo, but you can ask a bunch of people to compare two photos and say which looks straighter, then yo…
Yeah absolutely. In your link, it iterates on _ = ^{_}, until it finds the fixed point.
In our training pipeline, we had to convert the fixed point iteration to be on _ directly for numerical stability. I have a post on that here!: https://hackmd.io/x3_EkXGKRdeq-rNHo_RpZA
Bradley-Terry also very cleanly turns into a loss function that you can do gradient descent on, which will cause your model to efficiently learn Elo scores! Our calculations are at: https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw
Re: Show HN: Improving search ranking with chess Elo scores
#60Awesome! This is great! The link in the article to the full blog explaining rerankers is 404ing for me. Questions to you as an expert related to search ranking. With o3 and source quality thresholds when performing web search. Could we implement an ELO-style cutoff where systems default to “I don’t know” rather than citing low-ranked sources? Currently o3’s main weakness is mixing high-quality sources with poor ones…
Hey! Thanks so much!
I fixed the link thanks for flagging.
Yes the same approach could be used for internet search. The fact that we now have an "absolute score" is very interesting since we can also use a threshold value to determine when an answer simply doesn't exist in a corpus.
The only issue is that if all scores are below the cutoff value, you end up discarding them all, and end up with many "I don't know"s. Best approach could just be to flag the "trust" the model has in each source retrieved and use it as such.