Ask HN: How are people doing AI evals these days? • Hacker News Board

Re: Ask HN: How are people doing AI evals these days?

#31

Post by yelmahallawy » Thu, Mar 12, 2026, 7:07 AM UTC

fluffet wrote: post #27

It's kind of bespoke for me tbh. For a co-pilot inside an app that could answer product questions, I looked at ~2000 or so support emails. I asked one LLM to dig out "How would you formulate the users question into a chatbot-like question from this email thread" and "What is the actual answer that should be in the response from this email thread", then just asked our bot that question, and have another LLM rate the a…

This makes sense. I am particularly interested in your invoice processing app example because the accuracy of those outputs can be quantitatively measured from 0%-100% accuracy.

I'm curious as to what is _good enough_ and how many iterations it takes to get there. Is 100% the only acceptable threshold? If so, how many iterations does that take? What does that process look like? Okay let's say 100% accuracy is too difficult to reach, then how do you choose your minimum acceptable threshold (is 95% accuracy good enough? is 90%?). Do you have a dedicated set of outputs and documents used for evals? I'd love to hear more about this example (if you worked directly on the evals for this app).

Re: Ask HN: How are people doing AI evals these days?

#32

Post by yelmahallawy » Thu, Mar 12, 2026, 7:09 AM UTC

raw_anon_1111 wrote: post #20

One of my specialties is AWS Connect based call centers https://news.ycombinator.com/item?id=47241412 I use LLMs to determine what a caller’s “intent” is. I do my best with my initial prompt and then I have the “business” test it and I log phrases that they use. I then make those phrases my scripted test suite. Any changes in prompts or models get put through the same test suite. In my case, I give my customers a web…

Do you play with the temperature/top k parameters at all?

Re: Ask HN: How are people doing AI evals these days?

#33

Post by yelmahallawy » Thu, Mar 12, 2026, 7:15 AM UTC

kelseyfrog wrote: post #6

Automated benchmarking. We were lucky enough to have PMs create a set of questions, we did a round of generation and labeled pass/fail annotations on each response. From there we bootstrapped AI-as-a judge and approximately replicated the results. Then we plug in new models, change prompts, pipelines while being able to approximate the original feedback signal. It's not an exact match, but it's wildly better than one…

This is interesting approach, thanks for the insight! If I may ask, _approximately_ how long does it take to test a newly-released model with the current strategy?

Re: Ask HN: How are people doing AI evals these days?

#34

Post by yelmahallawy » Thu, Mar 12, 2026, 7:20 AM UTC

kbdiaz wrote: post #17

The vast majority of AI companies I talk to seem to evaluate models mostly based on vibes. At my company, we use a mix of offline and online evals. I’m primarily interested in search agents, so I’m fortunate that information retrieval is a well-developed research field with clear metrics, methodology, and benchmarks. For most teams, I recommend shipping early/dogfooding internally, collecting real traces, and then ha…

Ah, interesting – yeah only swapping out the model isn't super insightful since models perform differently given different prompts. I'm going to look into GEPA, thanks!

Re: Ask HN: How are people doing AI evals these days?

#35

Post by yelmahallawy » Thu, Mar 12, 2026, 7:22 AM UTC

tmaly wrote: post #28

I was thinking about something similar the other day. I have seen a repeating pattern of people complaining that a new model comes out, it's amazing for a few weeks, then they nerf it. Most of these claims are subjective. I was thinking if we had a standardized chain of though representation, and if we could capture each models chain of thought into this standardized format, we could compare these for the same tasks…

Yeah that's essentially what I'm looking for. Since now that AI has become such a core part of most businesses, it's pretty critical to use the _best_ models + prompts for whatever your use case is.

Re: Ask HN: How are people doing AI evals these days?

#36

Post by yelmahallawy » Thu, Mar 12, 2026, 7:24 AM UTC

minikomi wrote: post #16

The more you can afford to build up your understanding of the problem space and define what inputs & outputs look like, the more flexible you can be with evals. Unfortunately, this is a lot of work and requires thinking and discussion with your team and those involved. https://poyo.co/note/20260217T130137/ I wrote about general ideas I take towards simple single prompt features, but most of it is applicable to more i…

Ah good read, thanks for sharing!

Re: Ask HN: How are people doing AI evals these days?

#37

Post by yelmahallawy » Thu, Mar 12, 2026, 7:27 AM UTC

satisfice wrote: post #15

It’s called testing. And from the reports and comments, there doesn’t seem to be much of it happening. The reason is: it’s quite expensive to do well. I find that for every hypothesis I might have to run a thousand prompts to collect enough data for a conclusion. For instance, to discover how reliably different models can extract noun phrases from a text: hours of grinding. Even so that was for a small text. I haven’…

Yeah it's a super tedious process and I was hoping that _maybe_ there is a tool out there that can help with this.

Re: Ask HN: How are people doing AI evals these days?

#38

Post by yelmahallawy » Thu, Mar 12, 2026, 7:28 AM UTC

moltar wrote: post #23

I use Promptfoo

Any takeaways? Has it been helpful? OpenAI just acquired them so it's probably useful but I was curious to hear more from people who've actually used it.

Re: Ask HN: How are people doing AI evals these days?

#39

Post by yelmahallawy » Thu, Mar 12, 2026, 7:29 AM UTC

celestialcheese wrote: post #5

mix of promptfoo and ad-hoc python scripts, with langfuse observability. Definitely not happy with it, but everything is moving too fast to feel like it's worth investing in.

Any takeaways on Promptfoo?

Re: Ask HN: How are people doing AI evals these days?

#40

Post by yelmahallawy » Thu, Mar 12, 2026, 7:37 AM UTC

bisonbear wrote: post #4

assume you're referencing coding agents - I don't think people are. If they are, it's likely using - AI to evaluate itself (eg ask claude to test out its own skill) - custom built platform (I see interest in this space) I've actually been thinking about this problem a lot and am working on making a custom eval runner for your codebase. What would your usecase be for this?

I'd love to hear more about what you're working on (if you're open to sharing!).

I like to play with knowledge base powered chatbots but what's most useful to me (and probably my primary use case) is coding agents since I use CC every day. Recently I just heard about Minimax m2.5 which apparently is a pretty good coding agent (they say it's comparable to opus 4.6) but I haven't tried it yet — plus it'd take a lot of time to figure out whether it's better or not.