Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on Evals

raindrop.ai

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Thoughts on Evals

#11

Post by gregsadetsky »

I'm new/uninformed in this world, but I have an idea for an eval that I think has not been tried yet.

Can anyone direct me towards how to ... make one? At the most fundamental level, is it about having test questions with known, golden (verified, valid) answers, and asking different LLM models to find the answer, and comparing scores (how many were found to be correct)?

What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm? What are non-obvious gotchas?

Finally, any known/commonly used frameworks to do this, or any tooling that can call different LLMs would be enough?

Thanks!

Re: Thoughts on Evals

#12

Post by koakuma-chan »

gregsadetsky wrote: post #11

I'm new/uninformed in this world, but I have an idea for an eval that I think has not been tried yet. Can anyone direct me towards how to ... make one? At the most fundamental level, is it about having test questions with known, golden (verified, valid) answers, and asking different LLM models to find the answer, and comparing scores (how many were found to be correct)? What are "obvious" things that are important to…

> Can anyone direct me towards how to ... make one?

https://hamel.dev/blog/posts/evals/

> What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm?

LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data.

> Finally, any known/commonly used frameworks to do this, or any tooling that can call different LLMs would be enough?

https://github.com/vercel/ai

https://github.com/mattpocock/evalite

Re: Thoughts on Evals

#13

Post by gregsadetsky »

koakuma-chan wrote: post #12
gregsadetsky wrote: post #11

I'm new/uninformed in this world, but I have an idea for an eval that I think has not been tried yet. Can anyone direct me towards how to ... make one? At the most fundamental level, is it about having test questions with known, golden (verified, valid) answers, and asking different LLM models to find the answer, and comparing scores (how many were found to be correct)? What are "obvious" things that are important to…

> Can anyone direct me towards how to ... make one? https://hamel.dev/blog/posts/evals/ > What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm? LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data. > Finally, any known/commonly used frameworks to do this, or any tooling…

I'm very grateful! Thanks a lot

Re: Thoughts on Evals

#14

Post by CharlieDigital »

basket_horse wrote: post #8
CharlieDigital wrote: post #7

Earlier quoted context omitted.

I think in most cases, outside of pure AI providers or think AI wrappers, almost every team will realize more gains from focusing on their user domains and solving business problems versus fine tuning their prompts to eek out a 5% improvement here and there.

I don’t think you can use this as a blanket statement. For many use cases the last 5-10% is the difference between demoware and production.

If that were true, just switching to TOON would make your startup take off.

That is obviously not true because a 5% gain in LLM performance isn't going to make up for a bad product.

Re: Thoughts on Evals

#15

Post by moltar »

gregsadetsky wrote: post #11

I'm new/uninformed in this world, but I have an idea for an eval that I think has not been tried yet. Can anyone direct me towards how to ... make one? At the most fundamental level, is it about having test questions with known, golden (verified, valid) answers, and asking different LLM models to find the answer, and comparing scores (how many were found to be correct)? What are "obvious" things that are important to…

Take a look at promptfoo

Re: Thoughts on Evals

#16

Post by ncgl »

koakuma-chan wrote: post #12
gregsadetsky wrote: post #11

I'm new/uninformed in this world, but I have an idea for an eval that I think has not been tried yet. Can anyone direct me towards how to ... make one? At the most fundamental level, is it about having test questions with known, golden (verified, valid) answers, and asking different LLM models to find the answer, and comparing scores (how many were found to be correct)? What are "obvious" things that are important to…

> Can anyone direct me towards how to ... make one? https://hamel.dev/blog/posts/evals/ > What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm? LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data. > Finally, any known/commonly used frameworks to do this, or any tooling…

"LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data."

Is this true? I remember there being a randomization factor in weighing tokens to make the output more something, dont recall what

Obviously I'm not an Ai dev

Re: Thoughts on Evals

#17

Post by koakuma-chan »

ncgl wrote: post #16
koakuma-chan wrote: post #12

Earlier quoted context omitted.

> Can anyone direct me towards how to ... make one? https://hamel.dev/blog/posts/evals/ > What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm? LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data. > Finally, any known/commonly used frameworks to do this, or any tooling…

"LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data." Is this true? I remember there being a randomization factor in weighing tokens to make the output more something , dont recall what Obviously I'm not an Ai dev

In my experience, the response may not be exactly the same, but the difference is negligible.
Post reply on HN