just checked this one https://benchmarks.llmonitor.com/sally and all got it wrong..
Asking 60 LLMs a set of 20 questions
11–20 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#12> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me:
> Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
Re: Asking 60 LLMs a set of 20 questions
#13I find it fascinating how similar the answers from different models with similar parameter counts are. Seems like an indication that they’re all modelling similarly datasets to a similar degree of fidelity.
Re: Asking 60 LLMs a set of 20 questions
#14Re: Asking 60 LLMs a set of 20 questions
#15Would be great to include inference times as well
Re: Asking 60 LLMs a set of 20 questions
#16I get frustrated when I tell an LLM “reply only with x” and then rather than responding “x”, it still responds with “Sure thing! Here’s x” or some other extra words.
Re: Asking 60 LLMs a set of 20 questions
#17[deleted]
Re: Asking 60 LLMs a set of 20 questions
#18This is very cool. Sorry if I missed it (poked around the site and your GitHub repo), but is the script available anywhere for others to run? Would love to publish results of running this against a series of ~10-20 open-source models with different quantization levels using Ollama and a 192GB M2 Ultra Mac Studio: https://github.com/jmorganca/ollama#model-library
Thanks. I haven’t put it online yet, but will try to clean it (removing API keys & all) tonight/tomorrow and publish it
:-) that's awesome. Thanks! Nice work on this.
Re: Asking 60 LLMs a set of 20 questions
#19> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
Nondeterminism strikes again!
But yes, I would expect GPT-4 to get this right most of the time.
Re: Asking 60 LLMs a set of 20 questions
#20> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
The second version of the Sally prompt reported on the benchmark has GPT4 giving the correct answer:
> Sally has 3 brothers. Each of these brothers has 2 sisters. This means that there are 2 girls in the family, including Sally. Therefore, Sally has 1 sister.
The prompt:
> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Let's think step by step.
The only difference with the first version being the addition of the last sentence.