Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Caveman: Why use many token when few token do trick

github.com

Post reply on HN
Print view View original

211–220 of 396 posts

Re: Caveman: Why use many token when few token do trick

#211

Post by spacemanspiff01 »

Perz1val wrote: post #176
bjackman wrote: post #66

If this really works there would seem to be a lot of alpha in running the expensive model in something like caveman mode, and then "decompressing" into normal mode with a cheap model. I don't think it would be fundamentally very surprising if something like this works, it seems like the natural extension to tokenisation. It also seems like the natural path towards "neuralese" where tokens no longer need to correspond…

But it can't, we see models get larger and larger and larger models perform better. made such huge improvements, because it makes more text for the language model to process. Cavemanising (lossy compression) the output does it to the input as well.

but some tokens are not really needed? This is probably bad because it is mismatched with training set, but if you trained a model on a dataset removing all prepositions (or whatever caveman speak is), would you have a performance degradation compared to the same model trained on the same dataset without the caveman translation?

Re: Caveman: Why use many token when few token do trick

#212

Post by dTal »

staminade wrote: post #40
Rexxar wrote: post #22

Earlier quoted context omitted.

> Someone didn't get the memo that for LLMs, tokens are units of thinking. Where do you get this memo ? Seems completely wrong to me. More computation does not translate to more "thinking" if you compute the wrong things (ie things that contribute significantly to the final sentence meaning).

That’s why you need filler words that contribute little to the sentence meaning but give it a chance to compute/think. This is part of why humans do the same when speaking.

The LLM has no accessible state beyond its own output tokens; each pass generates a single token and does not otherwise communicate with subsequent passes. Therefore all information calculated in a pass must be encoded into the entropy of the output token. If the only output of a thinking pass is a dumb filler word with hardly any entropy, then all the thinking for that filler word is forgotten and cannot be reconstructed.

Re: Caveman: Why use many token when few token do trick

#213

Post by WarmWash »

teekert wrote: post #10

Idk I try talk like cavemen to claude. Claude seems answer less good. We have more misunderstandings. Feel like sometimes need more words in total to explain previous instructions. Also less context is more damage if typo. Who agrees? Could be just feeling I have. I often ad fluff. Feels like better result from LLM. Me think LLM also get less thinking and less info from own previous replies if talk like caveman.

In the regular people forums (twitter, reddit), you see endless complaints about LLMs being stupid and useless.

But you also catch a glimpse of how the author of the complaint communicates in general...

"im trying to get the ai to help with the work i am doing to give me good advice for a nice path to heloing out and anytim i askin it for help with doing this it's total trash i dunt kno what to do anymore with this dum ai is so stupid"

Re: Caveman: Why use many token when few token do trick

#216

Post by adam_patarino »

dostick wrote: post #157
adam_patarino wrote: post #108

[flagged]

How is your offering different from local ollama?

Its batteries included. No config.

We also fine tuned and did RL on our model, developed a custom context engine, trained an embedding model, and modified MLX to improve inference.

Everything is built to work with each other. So it’s more like an apple product than Linux. Less config but better optimized for the task.

Re: Caveman: Why use many token when few token do trick

#218

Post by Chance-Device »

JBrussee-2 wrote: post #208

Author here. A few people are arguing against a stronger claim than the repo is meant to make. As well, this was very much intended to be a joke and not research level commentary. This skill is not intended to reduce hidden reasoning / thinking tokens. Anthropic’s own docs suggest more thinking budget can improve performance, so I would not claim otherwise. What it targets is the visible completion: less preamble, le…

Sounds reasonable to me. I think this thread is just the way online discourse tends to go. Actually it’s probably better than average, but still sometimes disappointing.
Post reply on HN