121–130 of 270 posts
Earlier quoted context omitted.
I have seen enough of this dismissal to call it the "human would also" kneejerk reaction.
Maybe if we make it a common enough reaction then these researchers like these would adopt the bare minimum of scientific rigour and test the same thing on a human control group. Because as it is I think the reaction is clearly still too rare.
Earlier quoted context omitted.
Did you look at the examples? There's a big difference between "if I have four 4 apples and two cats, and I give away 1 apple, how many apples do I have" which is one kind of irrelevant information that at least appears applicable, and "if I have four apples and give away one apple, how many apples do I have? Also, did you know cats use their tails to help balance?", which really wouldn't confuse most humans.
As someone who has written and graded a lot of University exams, I'm sure a decent number of students would write the wrong answer to that. A bunch of students would write 5 (adding all the numbers). Others would write "3 apples and 2 cats", which is technically not what I'm looking for (but personally I would give full marks for, some wouldn't). Many students clear try to answer exams by pattern matching, and I've s…
But in general I do not think these models are claiming at being good at replicating the performance of a distracted or otherwise low performing pupil. I think they should be evaluated against humans who are capable of completing word problems containing context that is not inherently necessary to the math question. The reason those tests I mentioned use these word problems is that it's a way to evaluate someone's ability to think in abstract mathematical terms about everyday situations, which obviously involve lots of unimportant information the person must choose to consider or not.
tl;dr: I think a reasonably competent high school student could answer the apple and cat question, which is absolutely a reasonable bar for an LLM to clear. If university students are failing these questions, then they have not been taught test taking skills, which should be considered a mathematical failure just as unacceptable as that of the LLM, not a mitigating similarity for the latter.
Earlier quoted context omitted.
For extra distraction, make the facts incorrect. Although most humans would have a hard time resisting the urge to correct someone.
Up to ten Nobel laureates have been unveiled as being three ducks in a trenchcoat.
> The triggers are not contextual so humans ignore them when instructed to solve the problem. Do they? I've found humans to be quite poor at ignoring irrelevant information, even when it isn't about cats. I would have insisted on a human control group to compare the results with.
Did you read a single one of the examples? No human would be influenced by these.
Step 1: ask the LLM to strip the nonsensical parts from the problem statement. Step 2: feed that to the LLM.
Step 2: feed that to the training algorithm.
* in a way that the meaning of the data is not changed
Earlier quoted context omitted.
If asked verbally that would absolutely confuse some humans. Easily enough to triple the error rate for that specific question (granted, that's easier than the actual questions, but still). Even in a written test with time pressure it would probably still have a statistically significant effect
The problem with your reasoning is that some humans cannot solve the problem even without the irrelevant info about cats. We can easily cherry pick our humans to fit any hypothesis about humans, because there are dumb humans. The issue is that AI models which, on the surface, appear to be similar to the smarter quantile of humans in solving certain problems, become confused in ways that humans in that problem-solving…
Earlier quoted context omitted.
Did you look at the examples? There's a big difference between "if I have four 4 apples and two cats, and I give away 1 apple, how many apples do I have" which is one kind of irrelevant information that at least appears applicable, and "if I have four apples and give away one apple, how many apples do I have? Also, did you know cats use their tails to help balance?", which really wouldn't confuse most humans.
If asked verbally that would absolutely confuse some humans. Easily enough to triple the error rate for that specific question (granted, that's easier than the actual questions, but still). Even in a written test with time pressure it would probably still have a statistically significant effect
Earlier quoted context omitted.
If asked verbally that would absolutely confuse some humans. Easily enough to triple the error rate for that specific question (granted, that's easier than the actual questions, but still). Even in a written test with time pressure it would probably still have a statistically significant effect
Is the model thinking what is cat doing here? Then start thinking it is being tested?
Funny, I was using chatGPT to have a conversation with a friend that doesn't speak English the other day. At the end of one of my messages, I appended 'how is your cat?', which was completely dropped from the translated output. I guess I'm doing it wrong?
They already adjusted ChatGPT to that study. Unrelated trailing cat content is now ignored.