Asking 60 LLMs a set of 20 questions • Hacker News Board

Re: Asking 60 LLMs a set of 20 questions

#31

Post by klohto » Sat, Sep 09, 2023, 2:50 PM UTC

jasonjmcghee wrote: post #27

klohto wrote: post #25

Earlier quoted context omitted.

GPT4 with a custom prompt gives the best results for me for all of the questions https://chat.openai.com/share/4897d1ad-0a5c-418c-babb-0de482...

"This conversation may reflect the link creator’s Custom Instructions, which aren’t shared and can meaningfully change how the model responds."

I know, I was sharing the answer itself :) Here is the foundation I’m using https://x.com/nisten/status/1696229059183730833

Re: Asking 60 LLMs a set of 20 questions

#32

Post by croes » Sat, Sep 09, 2023, 2:51 PM UTC

Are these LLMs deterministic or is this comparison rather useless?

Re: Asking 60 LLMs a set of 20 questions

#33

Post by jasonjmcghee » Sat, Sep 09, 2023, 2:52 PM UTC

Gunnerhead wrote: post #16

I get frustrated when I tell an LLM “reply only with x” and then rather than responding “x”, it still responds with “Sure thing! Here’s x” or some other extra words.

I see GPT-4 add extra flavor on the end instead - completely ignore "only do this" or "don't do that", and respond as usual, then at the very end "oh whoops I didn't do what you asked sorry about that!"

Re: Asking 60 LLMs a set of 20 questions

#34

Post by [deleted] » Sat, Sep 09, 2023, 2:52 PM UTC

cypress66 wrote: post #30

Is it using the correct prompt format for the different models? You should show exactly the string that was sent to the LLM.

[deleted]

Re: Asking 60 LLMs a set of 20 questions

#35

Post by adrian_b » Sat, Sep 09, 2023, 2:52 PM UTC

sixQuarks wrote: post #9

Has anyone looked through all the responses and chosen any winners?

I have not looked at all responses, but I have found interesting that at the question about the sisters of Sally, initially all LLMs have provided wrong answers.

Nevertheless, when the initial prompt has been augmented with "Let's think step by step.", that helped 2 LLMs to give the right answer: GPT 4 and ReMM SLERP L2 13B.

So at least for this kind of problem, there were 2 clear winners. The fact that one of them is GPT 4, is not a surprise.

Re: Asking 60 LLMs a set of 20 questions

#36

Post by [deleted] » Sat, Sep 09, 2023, 2:54 PM UTC

jasonjmcghee wrote: post #26

Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...

[deleted]

Re: Asking 60 LLMs a set of 20 questions

#37

Post by jasonjmcghee » Sat, Sep 09, 2023, 2:55 PM UTC

jasonjmcghee wrote: post #26

Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...

(Note: I posted this before the full benchmark above and uses default temperature, not temperature 0, but point stands)

Example (CodeLlama-Instruct 7B)

---

Sure, here are two poems, one arguing for the use of Kubernetes and another arguing against it:

For:

Kubernetes, a mighty tool,

Managing clusters with ease,

Scaling apps with grace.

Against:

Kubernetes, a beast to tame,

Complexity lurks in its name,

Leading teams astray.

Re: Asking 60 LLMs a set of 20 questions

#38

Post by rootusrootus » Sat, Sep 09, 2023, 2:56 PM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

Interestingly, it took GPT4 three attempts to give me the correct answer. The first two times it basically said the same [logically inconsistent] thing and concluded that Sally had two sisters.

Re: Asking 60 LLMs a set of 20 questions

#39

Post by adrian_b » Sat, Sep 09, 2023, 2:59 PM UTC

apples_oranges wrote: post #11

just checked this one https://benchmarks.llmonitor.com/sally and all got it wrong..

Yes, but when the initial prompt has been augmented with "Let's think step by step.", that helped 2 LLMs to give the right answer: GPT 4 and ReMM SLERP L2 13B.

Re: Asking 60 LLMs a set of 20 questions

#40

Post by Terretta » Sat, Sep 09, 2023, 3:01 PM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

From the ChatGPT app, using GPT-4 model, using my own custom prompt for such puzzles: “You are better than Marilyn Vos Savant at solving brain teaser logic puzzles step by step. Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?”

Certainly! Let's break it down step by step:

1. Sally has 3 brothers.

2. Each of those brothers has two sisters. Since all the brothers share the same two sisters, we are essentially hearing about the same sisters each time, not accumulating more sisters with each brother mentioned.

3. We know that Sally is a sister to her brothers, so she is one of the two sisters each brother has.

By putting it all together, we find that there must be another sister. Therefore, Sally has one sister.

---

New chats with no history, no custom prompt, and no system prompt, first interaction in the chat this question: “Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?”

All Answers (not cherry picked):

Sally has 1 sister. In the description, Sally is counted as one of the two sisters that each brother has. This means the brothers have another sister apart from Sally.

. . .

Sally has 1 sister. In the description, it mentioned that "Each brother has 2 sisters." This implies that there is Sally and another sister, since we know that all the brothers share the same sisters.

. . .

Sally has one sister. Each of her brothers has two sisters, counting Sally and the other sister.

. . .

Sally has 1 sister. Here's how we figure it out:

Sally has 3 brothers, and each of these brothers has 2 sisters. Since the brothers share the same sisters, we know that the two sisters must be Sally and another girl. So, Sally has 1 sister.

. . .

I have gotten zero wrong answers from GPT-4 in OpenAI’s ChatGPT app.