Deep Dive into G-Eval: How LLMs Evaluate Themselves
1–7 of 7 posts
Re: Deep Dive into G-Eval: How LLMs Evaluate Themselves
#2Are there any llms in particular that work best with g-evals?
Re: Deep Dive into G-Eval: How LLMs Evaluate Themselves
#3Interesting overview, though I still wonder how stable G-Eval really is across different model families. Auto-CoT helps with consistency, but I’ve seen drift even between API versions of the same model.
Re: Deep Dive into G-Eval: How LLMs Evaluate Themselves
#4Are there any llms in particular that work best with g-evals?
I haven’t come across any research showing that a specific LLM consistently outperforms others for this. It generally works best with strong reasoning models that produce consistent outputs.
Re: Deep Dive into G-Eval: How LLMs Evaluate Themselves
#5Interesting overview, though I still wonder how stable G-Eval really is across different model families. Auto-CoT helps with consistency, but I’ve seen drift even between API versions of the same model.
That's true. Even small API or model version updates can shift evaluation behavior. G-Eval helps reduce that variance, but it doesn’t eliminate it completely. I think long-term stability will probably require some combination of fixed reference models and calibration datasets.
Re: Deep Dive into G-Eval: How LLMs Evaluate Themselves
#6Are there any llms in particular that work best with g-evals?
LLM Benchmark leaderboard for common evals sounds like a fun idea to me.
Re: Deep Dive into G-Eval: How LLMs Evaluate Themselves
#7We’ve tried geval but it hasn’t been super useful in practice. If we run the same input on the same model and same geval 10 times we get significantly different results, so you can’t really arrive at any conclusions based on the results.