61–70 of 85 posts
Earlier quoted context omitted.
I wonder if RLHF interfered with 9.11 (which could be interpreted as a date), preventing the model from considering it naturally. Wonder if the same thing would have happened with 9.12. What was your original prompt?
"What is bigger, 9.9 or 9.11?"
LLM Confabulation (Hallucination): https://github.com/lechmazur/confabulations/
LLM Step Game: https://github.com/lechmazur/step_game
LLM Thematic Generalization Benchmark: https://github.com/lechmazur/generalization
LLM Creative Story-Writing Benchmark: https://github.com/lechmazur/writing
Extended NYT Connections LLM Benchmark: https://github.com/lechmazur/nyt-connections/
and a couple more that I haven't updated very recently.
What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…
The other day I watched Gemini reasoning go in circles for ~30 seconds trying to determine if 9.11 was greater than 9.9. ---- "Realize a direct comparison of tenths is not enough: Wait, this is not right. In 9.9, the tenths digit is 9. In 9.11, the tenths digit is 1. This initial comparison is incorrectly stated. Let's re-examine. Re-examine tenths place comparison (correctly): In 9.9, the tenths digit is 9. In 9.11,…
I'm not a fan of these "gotchas" because they don't test for what we really care about. Like counting the number of R's in strawberry, many of these are character-counting or character manipulation problems which tokenization is not well-suited for. I'm sure an engineer could come up with a clever way to train for this, but that seems like optimizing for the wrong thing. IMO these questions go in the wrong direction.…
In fact, your final statement that these are tasks software should do rather than LLMs, is only proven to more people and made more clear by the prominence of these "gotchas"
As if the whole anti-intellectual hunt wasn’t enough, now PhD is a category implying holder of rote-knowledge at the highest level. I guess it is hopeless to fight this, but a PhD is 100x more about the apprenticeship and real-world training as a scientist than any accumulated knowledge beyond ones prior training. I know this is a rant, sorry, just so tired of the stupidity.
This is also a bit of a pet peeve of mine too. I surely learned a lot of "knowledge" during my PhD, but really what makes me an expert in my subject is less my ability to rattle off facts but 1. my ability to reason and understand the subject after a lot of experience working in it, and 2. the skills that I picked up during my PhD and afterwards. I can really only "know" so much, but the skills that I developed help…
Earlier quoted context omitted.
It feels like lot of the reasoning tokens go to waste on pure brute force approach - plugging in numbers and evaluating and comparing against the answer. "Nope, that didn't work, let's try 4 instead of 6 this time", etc. What if the reward function instead focuses on diversity of procedures within a token budged (10k - 20k tokens). I.e. RL rewards the model in trying different methods or generating different hypothes…
I would say that diversity isn't something that's easy to reenforce, but I do think it will occur as a natural consequence of optimizing for shorter chains of thought according to a wide variety of problems. Of course, the nature of the data may lead it to do brute force, but that can be fixed with clever fine tuning.
For diversity reward, my thinking is basically looking at reasoning tokens in latent space - taking semantic similarity between subsequent chains, and if they are extremely similar, penalizing it.
I'm not a fan of these "gotchas" because they don't test for what we really care about. Like counting the number of R's in strawberry, many of these are character-counting or character manipulation problems which tokenization is not well-suited for. I'm sure an engineer could come up with a clever way to train for this, but that seems like optimizing for the wrong thing. IMO these questions go in the wrong direction.…
- The paper has an example where the model reasons "I'm frustrated" and then produces an answer that it "knows is wrong". You wouldn't know it if you didn't examine the reasoning tokens.
- There are two examples were R1 often gets stuck "thinking forever"
If these failures happen on these questions, where else can happen? We'll start to find out soon enough.
My feeling is that a lot of challenge could come from the tokenizer used by the model, similar to r in strawberry problem.
There is likely a little bit of that, but I don't think that's the primary factor that makes these hard. If you look at the reasoning performed by R1 ("Show Thoughts" checkbox), it has no problem splitting and joining candidate words into letters, or even doing the letter-level manipulations. Just to give an example: Take a familiar brand name, seen along roads and highways in the United States. It has five letters,…
Were you able to find a substantial number of questions that do not fall into the letter countinh or word shuffling domsin - problems that are clearly unrelated to the fundamental tokenizer issue of modern LLMs? Otherwise, I would argue that your paper simply proves that the issue still exists.