Viewing profile — ruby314

Also wanted to address the confusion on the role of llama-3-8b and llama-3.1-8b. In the blog post we use these models as an example of an evaluator LLM. We select what is best for …

Re #3 - my bad, mixing terminology in my answer above. It’s the “base model” for the evaluator model (vs a fine tuned evaluator model). Just using the labeled Halubench dataset as …

and yeah, we fully acknowledge that our list of references is not comprehensive :) thanks again for sharing!

"An advantage of being aware of early papers is that it accumulates citations so you can often find good works in reverse citations." - absolutely. Thanks again for sharing interes…

Ty! - We generate contrast pairs (for this post, using gpt-4o) and do some post processing for quality (synthetic data). The impact of different types of contrast pairs is a contin…

Thanks for sharing the reference. It's definitely related! But the approach in that paper is different in a couple of very significant ways: 1. They train a classifier on top of th…

yes that's correct! we project an evaluator LLM's internal activations onto meaningful linear directions, derived from contrasting examples. the strongest connection is to LLM inte…