LLM Evals Are Just Tests. Why Are We Making This So Complicated?
cameronwestland.com
LLM Evals Are Just Tests. Why Are We Making This So Complicated?
1–3 of 3 posts
Re: LLM Evals Are Just Tests. Why Are We Making This So Complicated?
#2So, did the tests allow you to build a system that never confused existing features with new features? That seems like the problem statement, but I think I'm only seeing probabilistic testing.
Re: LLM Evals Are Just Tests. Why Are We Making This So Complicated?
#3So, did the tests allow you to build a system that never confused existing features with new features? That seems like the problem statement, but I think I'm only seeing probabilistic testing.
Never? No. Way less likely? Yes!
In dev we do 100 consistency checks and get green. In CI we do 10.