Language models still struggle with the concept of negation
quantamagazine.org
Language models still struggle with the concept of negation
1–10 of 172 posts
Re: Language models still struggle with the concept of negation
#2The stop words bit is a bit wtf. Someone didn't do their homework
Re: Language models still struggle with the concept of negation
#3I just typed the examples into chatgpt4 and it got all of them right
Re: Language models still struggle with the concept of negation
#4I just typed the examples into chatgpt4 and it got all of them right
Will AI skeptics ever get tired of being btfo by stochastic parrots?
Gary Marcus's Twitter timeline says no.
Re: Language models still struggle with the concept of negation
#5What animals don’t have paws or lay eggs, but have wings? Work it out, step by step.
ChatGPT (GPT-4):
Sure, let's go through this step by step:
First, you asked for animals that don't have paws. This would exclude mammals such as dogs, cats, bears, and so on, as these species have paws.
Second, you specified animals that don't lay eggs. This eliminates a wide range of animals including all birds, most reptiles, and some fish and insects, as they all lay eggs.
Finally, you asked for animals that have wings. Wings are typically found in birds, bats, and insects.
Given these three criteria, we can see that we're in a bit of a conundrum. Most winged creatures either have paws (like bats) or lay eggs (like birds and insects).
However, if we consider the term "wings" more broadly, there are some species of bats that give live birth (don't lay eggs) and technically don't have "paws" but rather winged limbs adapted for flight.
So, based on the constraints of your question, the best fit would be certain species of bats, provided we interpret "wings" in a broad sense and "paws" in a strict sense.
Re: Language models still struggle with the concept of negation
#6What animals don’t have paws or lay eggs, but have wings? Work it out, step by step. ChatGPT (GPT-4): Sure, let's go through this step by step: First, you asked for animals that don't have paws. This would exclude mammals such as dogs, cats, bears, and so on, as these species have paws. Second, you specified animals that don't lay eggs. This eliminates a wide range of animals including all birds, most reptiles, and s…
From what I’ve seen, there’s a huge difference in chat GTP 4 from other LLM
Re: Language models still struggle with the concept of negation
#7I find I have to state negations on what I want very emphatically in order for it to pick it up, like I'm carefully explaining to someone how to do something for the very first time in a way where they don't screw it up. I'll use stuff like MAKE SURE YOU DO THIS AND NOT THAT as a prompt, in all caps, and it seems to see that as emphasis.
I might also even have to repeat them in order for it to get things right but generally gpt-4 seems to pick up on these cues. It definitely feels like a sort of point where it could use some improvement because if I'm not extremely explicit it will give me some issues.
As an example: "Can you please recommend me 10 books about witches set in Europe that take place no later than the Napoleonic wars" wasn't enough of an exclusion to work, but "Hi! Can you please recommend me 10 books about witches set in Europe that take place no later than the Napoleonic wars? Please exclude any books that don't take place in Europe, and also exclude any books which take place after The Napoleonic Wars (1803–1815). Thanks, ChatGPT!" was exactly enough to return what we were looking for.
Re: Language models still struggle with the concept of negation
#8This reminded me of https://github.com/elsamuko/Shirt-without-Stripes
Whose days may be soon over as gpt4 gets this overall right..? As noted by many folks here
Re: Language models still struggle with the concept of negation
#9[deleted]
Re: Language models still struggle with the concept of negation
#10What animals don’t have paws or lay eggs, but have wings? Work it out, step by step. ChatGPT (GPT-4): Sure, let's go through this step by step: First, you asked for animals that don't have paws. This would exclude mammals such as dogs, cats, bears, and so on, as these species have paws. Second, you specified animals that don't lay eggs. This eliminates a wide range of animals including all birds, most reptiles, and s…
From what I’ve seen, there’s a huge difference in chat GTP 4 from other LLM
Interesting that it contradicted itself by saying bats have paws and don’t have paws, but arrived at the correct conclusion anyway.