41–50 of 67 posts
Fun fact about ELO. It's natural to think that it is some kind of initialism, but in fact ELO doesn't stand for anything. It's the name of the guy who invented the system. https://en.wikipedia.org/wiki/Arpad_Elo So don't say it "E.L.O." (unless you're talking about the band, I guess), say "ee-low"
Earlier quoted context omitted.
I was going to mention this approach as well. The problem with the OP is that it has assumption bias and the entire chain is based on that assumption. It’s novel. But the original idea was to more evenly distribute scores so you can find real relevance and I think 2AFC is better. But I don’t have time to verify and post a paper about it.
Yes our pairwise method is based entirely on 2AFC comparisons, for both intra-query and inter-query ELO calculations. It's definitely the best if not only way to get extremely high signal, and a score assignment that actually converges the more you sample. In terms of the "F" in 2AFC, we actually have this amusing snippet from our prompt: > Do NOT output a score of 0.0, ensure to focus on which document is superior,…
You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).
LambdaMART's approach seems better in that respect.
https://medium.com/@nikhilbd/pointwise-vs-pairwise-vs-listwi...
The link in the article to the full blog explaining rerankers is 404ing for me.
Questions to you as an expert related to search ranking. With o3 and source quality thresholds when performing web search. Could we implement an ELO-style cutoff where systems default to “I don’t know” rather than citing low-ranked sources?
Currently o3’s main weakness is mixing high-quality sources with poor ones when it uses the web search in the same response. The answer sounds authoritative throughout, but parts are backed by unreliable sources. This makes it harder to trust even the well-sourced portions (e.g. believing the US election is next year - not a hallucination but a poorly date formatted source it used). It also makes the response a lot slower.
Would a hard quality threshold be better than the current approach of seamlessly blending good and bad sources?
You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).
would you consider JS bindings? should be easy to vibe code given what you have. bonus points if it runs in the browser (eg export the wasm binary). thank you!
Fun fact about ELO. It's natural to think that it is some kind of initialism, but in fact ELO doesn't stand for anything. It's the name of the guy who invented the system. https://en.wikipedia.org/wiki/Arpad_Elo So don't say it "E.L.O." (unless you're talking about the band, I guess), say "ee-low"
Thanks for this :) I had never heard of Elo until I noticed this morning that the new Chess course in Duolingo gives you an Elo ranking after a few rounds against Oscar. Probably would have skipped right over this story and comments otherwise, but now I have a fun bit of non-tech trivia to share if it ever comes up in small talk someday.
Happy to see competition in rerankers! Good luck with your product. My questions: what languages do your models currently support? Did you perform multilingual benchmarks? Couldn't find an answer on the website