Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How are people doing AI evals these days?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Ask HN: How are people doing AI evals these days?

#41

Post by raw_anon_1111 »

yelmahallawy wrote: post #32
raw_anon_1111 wrote: post #20

One of my specialties is AWS Connect based call centers https://news.ycombinator.com/item?id=47241412 I use LLMs to determine what a caller’s “intent” is. I do my best with my initial prompt and then I have the “business” test it and I log phrases that they use. I then make those phrases my scripted test suite. Any changes in prompts or models get put through the same test suite. In my case, I give my customers a web…

Do you play with the temperature/top k parameters at all?

No. I also use the least sophisticated but fastest model that Amazon hosts - and it hosts all of them except OpenAI models - Nova Lite

Going from free text to tool call with parameters in the grand scheme of things is one of the easiest things to do especially when you only have a limited number of tools.

Re: Ask HN: How are people doing AI evals these days?

#42

Post by yelmahallawy »

maxalbarello wrote: post #7

Also wondering how to evals agentic pipelines. For instance, I generated memories from my chatGPT conversation history, how do I know whether they are accurate or not? I would like a single number that I would use to optimize the pipeline with but I find it hard to figure out what that number should be measuring.

And I think this is a common problem actually — figuring out what to measure and how to measure it – it's not black and white. What I do is have a few dimensions to measure it against (this may or may not fit your use case): relevance, instruction following, clarity, hallucination rate, etc. but even then, it becomes hard to measure things like 'clarity'.

Re: Ask HN: How are people doing AI evals these days?

#43

Post by yelmahallawy »

mierz00 wrote: post #24

I highly rate Braintrust. It wouldn’t be too difficult to build something like that for your own usage, but I found it pretty easy to get datasets set up. Essentially a game changer in understanding if your prompts are working. Especially if you’re doing something which requires high levels of consistency. In our case we would use LLM for classification which fits in perfectly with evals.

Have some good takeaways / feedback on this? First time I hear about Braintrust (the eval platform) so I'll look into it but I'm curious on your experience with it so far.

Re: Ask HN: How are people doing AI evals these days?

#44

Post by yelmahallawy »

gabdiax wrote: post #26

I'm also curious about this. In some cases I've seen teams rely on a mix of automated metrics and human review, especially for production systems where reliability matters a lot. But evaluation pipelines for AI still seem much less standardized compared to traditional software monitoring.

Yeah, it feels like an unsolved problem still. I've also seen many teams spend hours on human review in eval pipelines (and this accumulates with each new model that gets released).

Re: Ask HN: How are people doing AI evals these days?

#45

Post by yelmahallawy »

mock-possum wrote: post #21

We feed a handful of preset questions through the new AI, we collect the results, we ask another AI to score the answers based on example ‘hood’ answers we’ve written, then we have a guy sit down and use the fallout as a starting point to rank the performance of that AI, compared to all the previous ones. Seems like it works pretty well. Our prompts and params get tweaked towards better and better results, and we get…

The guy who reviews all of this, is his role in the company fully dedicated to reviewing these eval pipelines?

Re: Ask HN: How are people doing AI evals these days?

#46

Post by kelseyfrog »

yelmahallawy wrote: post #33
kelseyfrog wrote: post #6

Automated benchmarking. We were lucky enough to have PMs create a set of questions, we did a round of generation and labeled pass/fail annotations on each response. From there we bootstrapped AI-as-a judge and approximately replicated the results. Then we plug in new models, change prompts, pipelines while being able to approximate the original feedback signal. It's not an exact match, but it's wildly better than one…

This is interesting approach, thanks for the insight! If I may ask, _approximately_ how long does it take to test a newly-released model with the current strategy?

20mins or so. The bottleneck is rate-limiting. It's amenable to parallelization. Each tests can run in isolation at the same time.

Re: Ask HN: How are people doing AI evals these days?

#47

Post by yelmahallawy »

raw_anon_1111 wrote: post #41
yelmahallawy wrote: post #32

Earlier quoted context omitted.

Do you play with the temperature/top k parameters at all?

No. I also use the least sophisticated but fastest model that Amazon hosts - and it hosts all of them except OpenAI models - Nova Lite Going from free text to tool call with parameters in the grand scheme of things is one of the easiest things to do especially when you only have a limited number of tools.

Makes sense, simpler=better. Thanks!

Re: Ask HN: How are people doing AI evals these days?

#48

Post by yelmahallawy »

kelseyfrog wrote: post #46
yelmahallawy wrote: post #33

Earlier quoted context omitted.

This is interesting approach, thanks for the insight! If I may ask, _approximately_ how long does it take to test a newly-released model with the current strategy?

20mins or so. The bottleneck is rate-limiting. It's amenable to parallelization. Each tests can run in isolation at the same time.

Gotchu. Yeah that's pretty quick, awesome thanks!

Re: Ask HN: How are people doing AI evals these days?

#49

Post by moltar »

yelmahallawy wrote: post #38
moltar wrote: post #23

I use Promptfoo

Any takeaways? Has it been helpful? OpenAI just acquired them so it's probably useful but I was curious to hear more from people who've actually used it.

Yes, very useful. Can’t imagine managing a large interconnected prompt collection without it.

Re: Ask HN: How are people doing AI evals these days?

#50

Post by gabdiax »

yelmahallawy wrote: post #44
gabdiax wrote: post #26

I'm also curious about this. In some cases I've seen teams rely on a mix of automated metrics and human review, especially for production systems where reliability matters a lot. But evaluation pipelines for AI still seem much less standardized compared to traditional software monitoring.

Yeah, it feels like an unsolved problem still. I've also seen many teams spend hours on human review in eval pipelines (and this accumulates with each new model that gets released).

I’m building EventSentinel.ai, a predictive AI platform that monitors hardware and network infrastructure to detect early signals of failures and connectivity issues before they cause downtime.

I’m looking for a few early-stage design partners (SRE / DevOps / IT / Network teams) who:

Manage on‑prem or hybrid infrastructure with critical uptime requirements

Are currently using tools like Datadog, PRTG, Zabbix, or similar, but still deal with “surprise” incidents?

Are open to trying an MVP and giving candid feedback in short feedback sessions?

What you’d get:

-Early access to our predictive failure and anomaly detection features

-Direct influence on the roadmap based on your needs

-Free usage during the MVP phase (and preferential terms later)

If this sounds relevant, drop a comment “interested” and I’ll follow up with details or email at gabriele@eventsentinel.ai

Post reply on HN