Asking 60 LLMs a set of 20 questions • Hacker News Board

Re: Asking 60 LLMs a set of 20 questions

#21

Post by jakderrida » Sat, Sep 09, 2023, 2:33 PM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

Also, MPT 7B gets it right over half the time. I've been testing every new LLM with that question.

Also, I tend to include mention in the question that all siblings are from the same two parents to preclude half-siblings because half my friends have half-siblings from both sides scattered across the country; so the wrong answers actually do tend to apply to them sometimes.

Re: Asking 60 LLMs a set of 20 questions

#22

Post by rubymamis » Sat, Sep 09, 2023, 2:39 PM UTC

sixQuarks wrote: post #9

Has anyone looked through all the responses and chosen any winners?

GPT4 seems to me to be the best. Undi95/ReMM-SLERP-L2-13B the runner up.

Re: Asking 60 LLMs a set of 20 questions

#23

Post by broast » Sat, Sep 09, 2023, 2:41 PM UTC

Gunnerhead wrote: post #16

I get frustrated when I tell an LLM “reply only with x” and then rather than responding “x”, it still responds with “Sure thing! Here’s x” or some other extra words.

The curse of chat models.

Re: Asking 60 LLMs a set of 20 questions

#24

Post by cypress66 » Sat, Sep 09, 2023, 2:41 PM UTC

You should add what version of the model you are testing

For example you mention Jon Durbin Airoboros L2 70B

But is it 1.4? 2.0? 2.1? Etc.

Re: Asking 60 LLMs a set of 20 questions

#25

Post by klohto » Sat, Sep 09, 2023, 2:43 PM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

GPT4 with a custom prompt gives the best results for me for all of the questions https://chat.openai.com/share/4897d1ad-0a5c-418c-babb-0de482...

Re: Asking 60 LLMs a set of 20 questions

#26

Post by jasonjmcghee » Sat, Sep 09, 2023, 2:45 PM UTC

Where is that CodeLlama model from?

I've played around with it and instruct variant with dramatically better results than what is listed here.

I used Ollama.

Almost looks like weights were corrupted or something.

---

Update: My results using CodeLlama Instruct 7B, w/ Temperature 0

https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...

Re: Asking 60 LLMs a set of 20 questions

#27

Post by jasonjmcghee » Sat, Sep 09, 2023, 2:46 PM UTC

klohto wrote: post #25

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

GPT4 with a custom prompt gives the best results for me for all of the questions https://chat.openai.com/share/4897d1ad-0a5c-418c-babb-0de482...

"This conversation may reflect the link creator’s Custom Instructions, which aren’t shared and can meaningfully change how the model responds."

Re: Asking 60 LLMs a set of 20 questions

#28

Post by ftxbro » Sat, Sep 09, 2023, 2:47 PM UTC

llm has tainted for me every poem that ends a line as "with ease" or "without fear" i will always think those are made by llm from now on.

Re: Asking 60 LLMs a set of 20 questions

#29

Post by hombre_fatal » Sat, Sep 09, 2023, 2:47 PM UTC

klohto wrote: post #25

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

GPT4 with a custom prompt gives the best results for me for all of the questions https://chat.openai.com/share/4897d1ad-0a5c-418c-babb-0de482...

What was your custom prompt that bastardized GPT-4's response so badly with emojis and weird formatting?

Re: Asking 60 LLMs a set of 20 questions

#30

Post by cypress66 » Sat, Sep 09, 2023, 2:49 PM UTC

Is it using the correct prompt format for the different models? You should show exactly the string that was sent to the LLM.