Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

341–350 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#341

Post by labrador »

labrador wrote: post #191
drstewart wrote: post #185

Earlier quoted context omitted.

What's the point of inventing all these different materials? Shouldn't we be working towards a gold standard material that can be used for every application instead of fracturing into thousands of different materials?

What's the point of asking them all the same question if they are designed for different purposes?

You fine tune small models for different purposes. Small because big models are hardware intensive and for a special purpose you may not need the big model anyway. After you tune your small model for your special purpose, you ask it questions from a standard set to make sure you haven't tuned your model into incoherence.

Re: Asking 60 LLMs a set of 20 questions

#342

Post by lostmsu »

dontreact wrote: post #274
moffkalast wrote: post #193

Earlier quoted context omitted.

That's true, but those results are rarely the correct ones, at least for v1 llama models. In my experience each model has an optimal temperature at which it performs vastly better. I'm sure OpenAI have the best config they know set up for ChatGPT but let people generate trash through the API if they want to waste their credits on it.

Why would the accuracy decrease with lower temperature? Setting temperature to 0 just means at each step the model will emit the token with the highest likelihood.

I had a conversation with a friend regarding this exact question and my understanding is that model trains to optimize the distribution of all texts, therefore when you restrict it to deterministic sampling that is not representative of inputs you select the slice of the distribution that model learned that conveys much less information than the full distribution, and hence has poorer results.

Re: Asking 60 LLMs a set of 20 questions

#343

Post by MichaelMoser123 »

nonfamous wrote: post #339
MichaelMoser123 wrote: post #264

Earlier quoted context omitted.

google bard also gave the correct answer, even without adding 'lets think step by step'.

Google Bard is not a foundational LLM, it’s a chat app that draws on data from the web for grounding. It may have found the answer from a search result (perhaps this very page!).

it uses PaLM 2 - which is an LLM trained from scratch. https://en.wikipedia.org/wiki/PaLM

Re: Asking 60 LLMs a set of 20 questions

#344

Post by bicx »

typpo wrote: post #131

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…

I was just digging into promptfoo the other day for some good starting points in my own LLM eval suite. Thanks for the great work!

Re: Asking 60 LLMs a set of 20 questions

#346

Post by d4rkp4ttern »

ydant wrote: post #186
civilitty wrote: post #86

Earlier quoted context omitted.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

And it's only fixed for the stated case, but if you reverse the genders, GPT-4 gets it wrong. Q: Bobby (a boy) has 3 sisters. Each sister has 2 brothers. How many brothers does Bobby have? Let's think step by step. A: First, we know that Bobby has 3 sisters. Second, we know that each sister has 2 brothers. This means that Bobby has 2 brothers because the sisters' brothers are Bobby and his two brothers. So, Bobby has…

I just tried this with GPT4 (ChatGPT site), omitting the “step by step”, and it got it right

Re: Asking 60 LLMs a set of 20 questions

#348

Post by eazye711 »

typpo wrote: post #131

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…

Thanks for sharing, looks interesting!

I've actually been using a similar LLM evaluation tool called Arthur Bench: https://github.com/arthur-ai/bench

Some great scoring methods built in and a nice UI on top of it as well

Re: Asking 60 LLMs a set of 20 questions

#349

Post by agent_yellow_23 »

typpo wrote: post #131

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…

This is really cool!

I've been using this auditor tool that some friends at Fiddler created: https://github.com/fiddler-labs/fiddler-auditor

They went with a langchain interface for custom Evals which I really like. I am curious to hear if anyone has tried both of these. What's been your key take away for these?

Re: Asking 60 LLMs a set of 20 questions

#350

Post by nonfamous »

MichaelMoser123 wrote: post #343
nonfamous wrote: post #339

Earlier quoted context omitted.

Google Bard is not a foundational LLM, it’s a chat app that draws on data from the web for grounding. It may have found the answer from a search result (perhaps this very page!).

it uses PaLM 2 - which is an LLM trained from scratch. https://en.wikipedia.org/wiki/PaLM

And indeed, PalM 2 Bison is one of the LLMs covered in discussed link.

That doesn't change the fact that chat apps like Google Bard and Microsoft Bing, and even the ChatGPT app with plugins, are not foundational LLMs. The fact that they incorporate information from outside the LLM (usually in ways that are opaque to the user) gives them fundamentally different characteristics to LLMS.

Post reply on HN