About AI Evals
hamel.dev
About AI Evals
1–10 of 46 posts
Re: About AI Evals
#2[dead]
Re: About AI Evals
#3Some great info, but I have to disagree with this:
> Q: How much time should I spend on model selection?
> Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence. Does error analysis suggest that your model is the problem?”
If there's a clear jump in evals from one model to the next (ie Gemini 2 to 2.5, or Claude 3.7 to 4) that will level up your system pretty easily. Use the best models you can, if you can afford it.
Re: About AI Evals
#4Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
The vast majority of a I startups will fail for reasons other than model costs. If you crack your use case, model costs should fall exponentially.
Re: About AI Evals
#5Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
I might disagree as these models are pretty inscrutable, and behavior on your specific task can be dramatically different on a new/“better” model. Teams would do well to have the right evals to make this decision rather than get surprised.
Also the “if you can afford it” can be fairly non trivial decision.
Re: About AI Evals
#6Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
Yeah totally agree, I see so many systems perform badly only to find out they're using an older generation mode and simply updating to the current mode fixes many of their issues.
Re: About AI Evals
#7I'm biased in that I work on an open source project in this space, but I would strongly recommend starting with a free/open source platform for debugging/tracing, annotating, and building custom evals.
This niche of the field has come a very long way just over the last 12 months, and the tooling is so much better than it used to be. Trying to do this from scratch, beyond a "kinda sorta good enough for now" project, is a full-time engineering project in and of itself.
I'm a maintainer of Opik, but you have plenty of options in the space these days for whatever your particular needs are: https://github.com/comet-ml/opik
Re: About AI Evals
#8Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
I think the key part if that advice is the without evidence bit:
> I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence.
If you try to fix problems by switching from eg Gemini 2.5 Flash to OpenAI o3 but you don't have any evals in place how will you tell if the model switch actually helped?
Re: About AI Evals
#9Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
> If there's a clear jump in evals from one model to the next (ie Gemini 2 to 2.5, or Claude 3.7 to 4)
How do you know that their evals match behavior in your application? What if the older, "worse" model actually does some things better, but if you don't have comprehensive enough evals for your own domain, you simply don't know to check the things it's good at?
FWIW I agree that in general, you should start with the most powerful model you can afford, and use that to bootstrap your evals. But I do not think you can rely on generic benchmarks and evals as a proxy for your own domain. I've run into this several times where an ostensibly better model does no better than the previous generation.
Re: About AI Evals
#10> About AI Evals
Maybe it's obvious to some - but I was hoping that page started off by explaining what the hell an AI Eval specifically is.
I can probably guess from context but I'd love to have some validation.