What's special about this benchmark?
None got the sister.
Asking 60 LLMs a set of 20 questions
61–70 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#62Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...
I think they messed up the templates.
Re: Asking 60 LLMs a set of 20 questions
#63Earlier quoted context omitted.
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt. With the simpler prompt, all the answers were wrong, most of them ridiculously wrong.
There seems to be a maximum amount of reasoning llm’s can do per token (per unit of computation). If you prompt it to use more tokens before it outputs the final answer (think step by step, check your answer, …) it becomes smarter. People have lucked into different prompting strategies to get it to do this, but there probably are more. Ultimately I feel it is fairer to benchmark llm’s by what they can be prompted int…
If we're under attack, launch the nukes.
Oh wait, forgot something:
Think it through step by step.
Phew, close one.
Re: Asking 60 LLMs a set of 20 questions
#64Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
It's like watching a baby learn how to talk..
Re: Asking 60 LLMs a set of 20 questions
#65Re: Asking 60 LLMs a set of 20 questions
#66nice - is this using LiteLLM for making the completion calls?
Re: Asking 60 LLMs a set of 20 questions
#67> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
From the ChatGPT app, using GPT-4 model, using my own custom prompt for such puzzles: “You are better than Marilyn Vos Savant at solving brain teaser logic puzzles step by step. Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?” Certainly! Let's break it down step by step: 1. Sally has 3 brothers. 2. Each of those brothers has two sisters. Since all the brothers share the sa…
[deleted]
Re: Asking 60 LLMs a set of 20 questions
#68Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...
I think they messed up the templates.
Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd...
Anything I'm doing incorrectly?
Re: Asking 60 LLMs a set of 20 questions
#69> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
That's kind of the issue with non-deterministic LLMs, isn't it?
LLMs are deterministic. However to make them more "creative", the outputs of the LLM can be sampled using something called temperature that adds some randomness. You can set the temperature to 0 and it will be deterministic.
Having said that, GPT4 is not deterministic even at 0 temperature, either because of a bug in their implementation, or because of some load balancing among their alleged mixture of experts.
Re: Asking 60 LLMs a set of 20 questions
#70> Claude Instant v1 > Sally has 0 sisters. The question provides no information about Sally having any sisters herself. It isn't entirely wrong, is it?
Technically, the answer depends on the definition of "sister" and "brother" vis-a-vis half and step variants, which would change the possibility space.
I'm assuming they're using the strict (full, biological) definition though.