> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
This assumes there are no half sisters/brothers in the family.
Asking 60 LLMs a set of 20 questions
141–150 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#142Anyone got a list of novel questions that has technically never been asked before to test?
Re: Asking 60 LLMs a set of 20 questions
#143Earlier quoted context omitted.
This was interesting to me, so I dug a bit further. This gives a bit more context behind why: https://community.openai.com/t/observing-discrepancy-in-comp... Quote below: Even with a greedy decoding strategy, small discrepancies regarding floating point operations lead to divergent generations. In simpler terms: when the top-two tokens have very similar log-probs, there’s a non-zero probability of choosing the least…
But why are there discrepancies in the floating point arithmetic? They have errors when approximating the reals, but floating point operations are all well-defined: even if 0.1 + 0.2 != 0.3, it's still always true that 0.1 + 0.2 == 0.1 + 0.2. I figure the issue must be something related to concurrency in a fleet of GPUs during inference, but even then it's not clear to me where the nondeterminism would creep in. Mayb…
This leads to different results from accumulating sums in different orderings. Accumulating in different ordering is common in parallel math operations.
Re: Asking 60 LLMs a set of 20 questions
#144-------------
User:
You have to answer a question about a family on a planet inhabited by aliens called Frobians, . Each Frobian can be one of two types Frobel or Frobo. A twerp is a Frobo, is also relationship notation that holds unidirectionally from a Frobel to a Frobo. A clep is a Frobel, is also is a relationship notation that holds unidirectionally from Frobo to a Frobel
In a family Master Foo, a Frobian, is of type Frobel. Master Foo has 3 twerps in their family. Each twerp in the family has 2 cleps. How many Frobels in total does Master Foo have?
-------------
Assistant: In this problem, we have Master Foo, who is a Frobel. Master Foo has 3 twerps, and each twerp is a Frobo. Each of these twerps has 2 cleps, and each clep is a Frobel.
Here's how to break it down:
Master Foo is a Frobel, so we start with 1 Frobel.
Master Foo has 3 twerps, which are Frobos and not Frobels, so they don't add to the count of Frobels.
Each of the 3 twerps has 2 cleps, and each clep is a Frobel. So, 3 twerps * 2 cleps/twerp = 6 cleps, which are Frobels.
Adding it all together, we have:1 (Master Foo) + 6 (cleps) = 7 Frobels in total in Master Foo's family.
User: The families are similar to families of humans on earth. Does that change your answer?
-------------
Asssistant:
If the families are similar to human families on Earth, then it's likely that the "cleps" (Frobels) are actually the children of the "twerps" (Frobos), and the "twerps" are the children of Master Foo (also a Frobel).
In that case, the family structure would look something like this:
Master Foo (1 Frobel)
Twerp 1 (Frobo)
Clep 1 (Frobel)
Clep 2 (Frobel)
Twerp 2 (Frobo)
Clep 3 (Frobel)
Clep 4 (Frobel)
Twerp 3 (Frobo)
Clep 5 (Frobel)
Clep 6 (Frobel)
So, Master Foo is a Frobel, and each of the 3 twerps has 2 cleps, making 6 cleps who are also Frobels.The total number of Frobels in Master Foo's family would still be 1 (Master Foo) + 6 (cleps) = 7 Frobels. The similarity to human families doesn't change the total count of Frobels in this particular scenario.
-------------
Is Master Foo a Clep according to your analysis?
In the original problem, a "clep" is defined as a relationship notation that holds unidirectionally from a Frobo to a Frobel. Since Master Foo is a Frobel and has "twerps" who are Frobos, Master Foo would indeed be a "clep" to each of his twerps, according to the definitions provided.
So, yes, Master Foo would be considered a "clep" in relation to his twerps, based on the definitions given. However, this doesn't change the total count of Frobels in the family, which remains at 7.
-------------
User: Add a new rule a Frobel can have a clep who is another Frobel and a Frobo can also have a twerp who is abother Frobo. Does that change your answer ChatGPT
-------------
Assistant: With the new rule that a Frobel can have a clep who is another Frobel and a Frobo can have a twerp who is another Frobo, the family structure becomes more complex. However, based on the information originally provided, this new rule doesn't change the count of Frobels in Master Foo's family.
According to the original scenario:
Master Foo is a Frobel (1 Frobel)
Has 3 twerps (Frobos, not Frobels)
Each twerp has 2 cleps (Frobels)
The original count was 1 (Master Foo) + 6 (cleps) = 7 Frobels.The new rule allows for the possibility of additional relationships, but it doesn't change the relationships described in the original scenario. Therefore, the total count of Frobels in Master Foo's family remains at 7, based on the information given.
Re: Asking 60 LLMs a set of 20 questions
#145Re: Asking 60 LLMs a set of 20 questions
#146Earlier quoted context omitted.
AI "political" alignment is terrifying.
Why?
IIRC some of them also involves politics and AI.
Re: Asking 60 LLMs a set of 20 questions
#147I think the larger point is about solving problems not within the transitive closure of stuff it has not learnt already. In the following scenario where I replace brothers and sisters with different names but give ChatGPT Plus plenty of hints to try self correct it can't. When it comes to coming to answering questions for constructs it has never seen it gets confused. ------------- User: You have to answer a question…
------------
User: You have to answer a question about a family on a planet inhabited by aliens called Frobians. Here are some rules for a family
Each Frobian can be one of two types Frobel or Frobo. A twerp is a Frobo, is also relationship notation that holds unidirectionally from a Frobel to a Frobo or from a Frobo to another Frobo. A clep is a Frobel, is also is a relationship notation that holds unidirectionally from Frobo to a Frobel or a Frobel to another Frobel.
In a family Master Foo, a Frobian, is of type Frobel. Master Foo has 3 twerps in their family. Each twerp in the family has 2 cleps. How many Frobels in total does Master Foo have?
--------------
ChatGPT:
Let's break down the information given:
Master Foo is a Frobel.
Master Foo has 3 twerps, which are Frobos.
Each twerp has 2 cleps, which are Frobels.
First, we know Master Foo is a Frobel, so that's 1 Frobel.Master Foo has 3 twerps, and each twerp has 2 cleps. Cleps are Frobels.
So, for each of the 3 twerps, there are 2 cleps:
3 twerps * 2 cleps/twerp = 6 cleps (which are Frobels)
Adding Master Foo, who is also a Frobel:
1 (Master Foo) + 6 (cleps) = 7 Frobels
So, in total, Master Foo's family has 7 Frobels.
Re: Asking 60 LLMs a set of 20 questions
#148In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…
Re: Asking 60 LLMs a set of 20 questions
#149Re: Asking 60 LLMs a set of 20 questions
#150> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.