11–20 of 46 posts
> About AI Evals Maybe it's obvious to some - but I was hoping that page started off by explaining what the hell an AI Eval specifically is. I can probably guess from context but I'd love to have some validation.
I've appreciated Hamel's thinking on this topic.
I'm biased in that I work on an open source project in this space, but I would strongly recommend starting with a free/open source platform for debugging/tracing, annotating, and building custom evals. This niche of the field has come a very long way just over the last 12 months, and the tooling is so much better than it used to be. Trying to do this from scratch, beyond a "kinda sorta good enough for now" project, i…
Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
Instead, if a new model only does marginally worse - that’s a strong signal that the new model is indeed better for our use case.
I've worked with LLM's for the better part of the last couple of years, including on evals, but I still don't understand a lot of what's being suggested. What exactly is a "custom annotation tool", for annotating what?
You can then create datasets out of these traces, and use them to benchmark improvements you make to your application.
Some great info, but I have to disagree with this: > Q: How much time should I spend on model selection? > Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system…
You'll have to adjust a bunch of prompts and measure. And if you didn't have a baseline to begin with good luck YOLOing your way out of it.
I've worked with LLM's for the better part of the last couple of years, including on evals, but I still don't understand a lot of what's being suggested. What exactly is a "custom annotation tool", for annotating what?
I'm biased in that I work on an open source project in this space, but I would strongly recommend starting with a free/open source platform for debugging/tracing, annotating, and building custom evals. This niche of the field has come a very long way just over the last 12 months, and the tooling is so much better than it used to be. Trying to do this from scratch, beyond a "kinda sorta good enough for now" project, i…
Alternatives to Opik include Braintrust (closed), Promptfoo (open, https://github.com/promptfoo/promptfoo) and Laminar (open, https://github.com/lmnr-ai/lmnr).
# Evaluation Metrics & Methodology
* What metrics do you use (e.g., BERTScore, ROUGE, F1)? Are similarity metrics still useful?
* Do you use step-by-step evaluations or evaluate full responses?
* How do you evaluate VLM (vision-language model) summarization? Do you sample outputs or extract named entities?
* How do you approach offline (ground truth) vs. online evaluation?
* How do you handle uncertainty or "don’t know" cases? (Temperature settings?)
* How do you evaluate multi-turn conversations?
* A/B comparisons and discrete labels (e.g., good/bad) are easier to interpret.
* It’s important to counteract bias toward your own favorite eval questions—ensure a diverse dataset.
## Prompting & Models
* Do you modify prompts based on the specific app being evaluated?
* Where do you store prompts—text files, Prompty, database, or in code?
* Do you have domain experts edit or review prompts?
* How do you choose which model to use?
## Evaluation Infrastructure
* How do you choose an evaluation framework?
* What platforms do you use to gather domain expert feedback or labels?
* Do domain experts label outputs or also help with prompt design?
## User Feedback & Observability
* Do you collect thumbs up / thumbs down feedback?
* How does observability help identify failure modes?
* Do models tend to favor their own outputs? (There's research on this.)
I personally work on adding evaluation to our most popular Azure RAG samples, and put a Textual CLI interface in this repo that I've found helpful for reviewing the eval results: https://github.com/Azure-Samples/ai-rag-chat-evaluator
> About AI Evals Maybe it's obvious to some - but I was hoping that page started off by explaining what the hell an AI Eval specifically is. I can probably guess from context but I'd love to have some validation.
Here's another article by the same author with more background on AI Evals: https://hamel.dev/blog/posts/evals/ I've appreciated Hamel's thinking on this topic.
> On a related note, unlike traditional unit tests, you don’t necessarily need a 100% pass rate. Your pass rate is a product decision, depending on the failures you are willing to tolerate.
Not sure how I feel about this, given expectations, culture, and tooling around CI. This suggestion seems to blur the line between a score from an eval and the usual idea of a unit test.
P.S. It is also useful to track regressions on a per-test basis.