Ask HN: How are people doing AI evals these days?

Post by mierz00 » Tue, Mar 17, 2026, 7:57 AM UTC

yelmahallawy wrote: post #43

mierz00 wrote: post #24

I highly rate Braintrust. It wouldn’t be too difficult to build something like that for your own usage, but I found it pretty easy to get datasets set up. Essentially a game changer in understanding if your prompts are working. Especially if you’re doing something which requires high levels of consistency. In our case we would use LLM for classification which fits in perfectly with evals.

Have some good takeaways / feedback on this? First time I hear about Braintrust (the eval platform) so I'll look into it but I'm curious on your experience with it so far.

If I am being honest, the value came from doing evals and testing against different models.

Essentially all I needed was a way to upload a data set, run tests against that data set and spit out a percentage of pass fail.

Braintrust makes this pretty easy, but If I was to do it again I would vibecode the same functionality.

Post by mock-possum » Tue, Mar 17, 2026, 3:37 PM UTC

yelmahallawy wrote: post #45

mock-possum wrote: post #21

We feed a handful of preset questions through the new AI, we collect the results, we ask another AI to score the answers based on example ‘hood’ answers we’ve written, then we have a guy sit down and use the fallout as a starting point to rank the performance of that AI, compared to all the previous ones. Seems like it works pretty well. Our prompts and params get tweaked towards better and better results, and we get…

The guy who reviews all of this, is his role in the company fully dedicated to reviewing these eval pipelines?

Yeah - and he’s kind of a black box of a contractor, people kept saying his name, which is unusual, and at first I figured it was some software or other company we were using - eventually I realized it was a real guy, who we just feed LLM results to and he ranks them for us. He’s not a full time employee and I’ve never actually seen him or had any contact with him, so now I think it’s entertaining to imagine that he’s a figment of the CEO’s imagination - his alter ego that takes over after hours and obsessively reviews LLM outputs.

Re: Ask HN: How are people doing AI evals these days?

Re: Ask HN: How are people doing AI evals these days?