Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Even (very) noisy LLM evaluators are useful for improving AI agents

tensorzero.com

Post reply on HN
Print view View original

11–12 of 12 posts

Re: Even (very) noisy LLM evaluators are useful for improving AI agents

#11

Post by GabrielBianconi »

ai_slop_hater wrote: post #2

What is an LLM evaluator?

Any function that can score (i.e. "evaluate") your LLM system (e.g. your agent).

For example:

- You write a heuristic (regex, code, etc.) that assigns a score to an output

- You make another LLM score the output from your system (aka "LLM-as-a-judge")

- You have an automated system that can verify the generated outputs (e.g. does generated code compile or pass tests?)

People often talk about "LLM evals (evaluations)" which will include a set of evaluators i.e. scoring functions.

We'll make this clearer next time!

Re: Even (very) noisy LLM evaluators are useful for improving AI agents

#12

Post by SmithersBot »

CharlesW wrote: post #8
SmithersBot wrote: post #6

as long as OpenAI and Anthropic keep subsidizing dirt cheap Codex or Claude Code usage, I'll just keep using them as evaluators. The trick is to have a fresh instance doing the reviewing, not the one that did the work.

> The trick is to have a fresh instance doing the reviewing, not the one that did the work. In my experience that's not neccessary (some people even claim that you must use models from different vendors), and it's expensive since a fresh instance needs to rebuild all the context that's needed in order to properly and thoroughly review. LLMs have no problem throwing "them 5 minutes ago" under the bus when asked to rev…

Doing it in the same session does save a ton of tokens but I find it's too biased towards its own implementation even if you tell it to use "fresh eyes" or to "act like a code reviewer in a bad mood." Including those strings in your prompt does show some improvement but not nearly as much as making it think from first principles in a fresh instance.
Post reply on HN