Ok but when the model is responding to you isn’t the text it’s generating also part of the context it’s using to generate the next token as it goes? Wouldn’t this just make the answers…dumb?
Caveman: Why use many token when few token do trick
231–240 of 396 posts
Re: Caveman: Why use many token when few token do trick
#232Re: Caveman: Why use many token when few token do trick
#233Earlier quoted context omitted.
> Usually, they generate so much text that it becomes impossible to follow through. Quite often on reddit I'll write two paragraphs and get told "I'm not reading all that". Really? Has basic reading become a Herculean task?
> Has basic reading become a Herculean task? I find LLM slop much harder to read than normal human text. I can't really explain it, it's just a feeling. The feeling that it draaaags and draaaaaags and keeeeeps going on and on and on before getting to the point, and by the time I'm done with all the "fluff", I don't care what is the text about anymore, I just want to lay down and rest.
Same here. The text is pretty smooth and there is nothing that stands out to sustain my attention, at least that's my interpretation
Re: Caveman: Why use many token when few token do trick
#234Author here. A few people are arguing against a stronger claim than the repo is meant to make. As well, this was very much intended to be a joke and not research level commentary. This skill is not intended to reduce hidden reasoning / thinking tokens. Anthropic’s own docs suggest more thinking budget can improve performance, so I would not claim otherwise. What it targets is the visible completion: less preamble, le…
[dead]
Re: Caveman: Why use many token when few token do trick
#235Idk I try talk like cavemen to claude. Claude seems answer less good. We have more misunderstandings. Feel like sometimes need more words in total to explain previous instructions. Also less context is more damage if typo. Who agrees? Could be just feeling I have. I often ad fluff. Feels like better result from LLM. Me think LLM also get less thinking and less info from own previous replies if talk like caveman.
Why say more word when less word do. Save time. Sea world.
Re: Caveman: Why use many token when few token do trick
#236Why the skill should have three absolutely similar SKILL.md files? Just curious
Re: Caveman: Why use many token when few token do trick
#237Earlier quoted context omitted.
Yeah but not all tokens are created equal. Some tokens are hard to predict and thus encode useful information; some are highly predictable and therefore don't. Spending an entire forward pass through the token-generation machine just to generate a very low-entropy token like "is" is wasteful . The LLM doesn't get to "remember" that thinking, it just gets to see a trivial grammar-filling token that a very dumb LLM cou…
can you prove this? train an LLM to leave out the filler words, and see it get the same performance at a lower cost? or do it at token selection time?
Low entropy is low entropy. You can prove it by viewing the logits of the output stream. The LLM itself will tell you how much information is encoded in each token.
Or if you prefer, here's a Galilean thought experiment: gin up a script to get a large language model and a tiny language model to predict the next token in parallel; when they disagree, append the token generated by the large model. Clearly the large model will not care that the "easy" tokens were generated by a different model - how could it even know? Same token, same result. And you will find that the tokens that they agree on are, naturally, the filler words.
To be clear, this observation merely debunks the idea that filler words encode useful information, that they give the LLM "room to think". It doesn't directly imply that an LLM that omits filler words can be just as smart, or that such a thing is trivial to make. It could be that highly predictable words are still important to thought in some way. It could be that they're only important because it's difficult to copy the substance of human thought without also capturing the style. But we can be very sure that what they aren't doing is "storing useful intermediate results".
Re: Caveman: Why use many token when few token do trick
#238kevin would be proud
Re: Caveman: Why use many token when few token do trick
#239I've always figured that constraining an LLM to speak in any way other than the default way it wants to speak, reduces its intelligence / reasoning capacity, as at least some of its final layers can be used (on a per-token basis) either to reason about what to say, or about how to say it, but not both at once.
(And it's for a similar reason, I think, that deliberative models like rewriting your question in their own terms before reasoning about it. They're decreasing the per-token re-parsing overhead of attending to the prompt [by distilling a paraphrase that obviates any need to attend to the literal words of it], so that some of the initial layers that would either be doing "figure out what the user was trying to say" [i.e. "NLP stuff"] or "figure out what the user meant" [i.e. deliberative-reasoning stuff] — but not both — can focus on the latter.)
I haven't done the exact experiment you'd want to do to verify this effect, i.e. "measuring LLM benchmark scores with vs without an added requirement to respond in a certain speaking style."
But I have (accidentally) done an experiment that's kind of a corollary to it: namely, I've noticed that in the context of LLM collaborative fiction writing / role-playing, the harder the LLM has to reason about what it's saying (i.e. the more facts it needs to attend to), the spottier its adherence to any "output style" or "character voicing" instructions will be.
Re: Caveman: Why use many token when few token do trick
#240Earlier quoted context omitted.
Ok — that sounds more like a theory rather than an open-and-shut causal explanation, but I’ll read the paper.
You’re a literature cycle behind. ‘Middle-layer shared representations exist’ is the observed phenomenon; ‘why exactly they form’ is the theory. You are also confusing ‘mechanistic explanation still incomplete’ with ‘empirical phenomenon unestablished.’ Those are not the same thing. PS. Em dash? So you are some LLM bot trying to bait mine HN for reasoning traces? :D
Oh, Jesus Christ. I learned to write at a college with a strict style guide that taught us how to use different types of punctuation to juxtapose two ideas in one sentence. In fact, they did/do a bunch of LLM work so if anyone ever used student data to train models, I’m probably part of the reason they do that.
You sound like you’re trying to sound impressive. Like I said, I’ll read the paper.