[dead]
Ask HN: How are people doing AI evals these days?
11–20 of 52 posts
Re: Ask HN: How are people doing AI evals these days?
#12[deleted]
Re: Ask HN: How are people doing AI evals these days?
#13[flagged]
Re: Ask HN: How are people doing AI evals these days?
#14[dead]
Re: Ask HN: How are people doing AI evals these days?
#15It’s called testing. And from the reports and comments, there doesn’t seem to be much of it happening. The reason is: it’s quite expensive to do well.
I find that for every hypothesis I might have to run a thousand prompts to collect enough data for a conclusion. For instance, to discover how reliably different models can extract noun phrases from a text: hours of grinding. Even so that was for a small text. I haven’t yet run the process on a large text.
Re: Ask HN: How are people doing AI evals these days?
#16The more you can afford to build up your understanding of the problem space and define what inputs & outputs look like, the more flexible you can be with evals. Unfortunately, this is a lot of work and requires thinking and discussion with your team and those involved.
https://poyo.co/note/20260217T130137/
I wrote about general ideas I take towards simple single prompt features, but most of it is applicable to more involved agentic approaches too.
Re: Ask HN: How are people doing AI evals these days?
#17The vast majority of AI companies I talk to seem to evaluate models mostly based on vibes.
At my company, we use a mix of offline and online evals. I’m primarily interested in search agents, so I’m fortunate that information retrieval is a well-developed research field with clear metrics, methodology, and benchmarks. For most teams, I recommend shipping early/dogfooding internally, collecting real traces, and then hand-curating a golden dataset from those traces.
Many people run simple ablation experiments where they swap out the model and see which one performs best. That approach is reasonable, but I prefer a more rigorous setup.
If you only swap the model, some models may appear to perform better simply because they happen to work well with your prompt or harness. To avoid that bias, I use GEPA to optimize the prompt for each model/tool/harness combination I’m evaluating.
Re: Ask HN: How are people doing AI evals these days?
#18Doing tickets and commenting cost and quality in the PR.
Still, the best are outstanding, and the medium ones bare usable. I rank it by IQ. From 140 to utterly stupid. opencode/gpt-oss-120b local got a 90. opencode/opus-4.6 gets 140. codex/gpt-5.4 gets 115. All for C/C++ tasks.
There was one expensive Chinese SWE benchmark posted recently to arxiv. It did confirm my evaluation.
Re: Ask HN: How are people doing AI evals these days?
#19[flagged]
What you've said is nothing to do with AI Evals.
Re: Ask HN: How are people doing AI evals these days?
#20One of my specialties is AWS Connect based call centers
https://news.ycombinator.com/item?id=47241412
I use LLMs to determine what a caller’s “intent” is. I do my best with my initial prompt and then I have the “business” test it and I log phrases that they use.
I then make those phrases my scripted test suite. Any changes in prompts or models get put through the same test suite. In my case, I give my customers a website they can use to test new prompts and takes care of versioning.
I also log phrases that didn’t trigger an intent and modify the prompt and put it back through the suite.