Building the Ed-O-Meter: Notes on Writing My Own LLM Benchmark

Author here. This began as a weekend project because I didn't trust the public leaderboards — as what I was experiencing seemed to differ from the professional benchmarkers. I really wanted to see how the latest LLMs worked on 'my realworld' tasks. A mix of random things like planning a holiday, getting a vegetarian recipe, debugging code, jailbreak attempts. The results are here if you want to poke at them: https://reinvently.co.uk/tools/ed-o-meter/

So I suppose a slightly more scientific way of doing my usual vibechecks, that I can rerun regularly if I want to.

The write-up is more about the mistakes than the scores: e.g. figuring out how to treat refusals, dealing with false negatives when trying to measure results in a deterministic way. I learned quickly why most people don't do this (it's harder than it looks), but also gained some practical understanding of the nuances of these LLMs. Whilst the market sees the commoditisation of the capabilities, the behaviours of the models are diverging, making them less interchangeable if you want the optimal results.

The framework is pretty tight, only about 1k lines of python, it's shared on GitHub in-case anyone wants to have a go at building their own Eval suite: https://github.com/ed-is-ai/featherbench. It's designed to be easy to integrate and adapt to any python project where you just want some simple, bespoke Evals.

neat! I will check it out. I'm not really using local LLMs as dont have a big enough machine for these to run with a decent tps