Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Caveman: Why use many token when few token do trick

github.com

Post reply on HN
Print view View original

301–310 of 396 posts

Re: Caveman: Why use many token when few token do trick

#301

Post by andsoitis »

dTal wrote: post #251
padolsey wrote: post #170

This is fun. I'd like to see the same idea but oriented for richer tokens instead of simpler tokens. If you want to spend less tokens, then spend the 'good' ones. So, instead of saying 'make good' you could say 'improve idiomatically' or something. Depends on one's needs. I try to imagine every single token as an opportunity to bend/expand/limit the geometries I have access to. Language is a beautiful modulator to ap…

Hmm... this sounds a lot like the old RISC vs CISC argument all over again. RISC won because simplicity scales better and you can always define complex instructions in terms of simple ones. So while I would relish experiencing the timeline in which our computerized chums bootstrap into sentience through the judicious application of carefully selected and highly nuanced words, it's playing out the other way: LLMs doin…

RISC good. CISC bad. But CISC tribe sneaky — hide RISC inside. Look CISC outside, think RISC inside. Trick work long time.

Then ARM come. ARM very RISC. ARM go in phone. ARM go in tablet. ARM go everywhere. Apple make ARM chip, beat x86 with big club. Many impressed. Now ARM take server too. x86 tribe scared.

RISC-V new baby RISC. Free for all. Many tribe use. Watch this one.

RISC win brain fight. x86 survive by lying. ARM win world.

Re: Caveman: Why use many token when few token do trick

#302

Post by dTal »

krackers wrote: post #282
dTal wrote: post #202

Earlier quoted context omitted.

Yeah but not all tokens are created equal. Some tokens are hard to predict and thus encode useful information; some are highly predictable and therefore don't. Spending an entire forward pass through the token-generation machine just to generate a very low-entropy token like "is" is wasteful . The LLM doesn't get to "remember" that thinking, it just gets to see a trivial grammar-filling token that a very dumb LLM cou…

>They aren't stenographically hiding useful computation state in words like "the" and "and". When producing a token the model doesn't just emit the final token but you also have the entire hidden states from previous attention blocks. These hidden states are mixed into the attention block of future tokens (so even though LLMs are autoregressive where a token attends to previous tokens, in terms of a computational gra…

Huh okay, there was a major gap in my mental model. Thanks for helping to clear it up.

Re: Caveman: Why use many token when few token do trick

#304

Post by pxc »

reedlaw wrote: post #289
pxc wrote: post #276

Earlier quoted context omitted.

> Are you saying Chinese is more concise than English? Yeah, definitely. It lacks case and verb conjugations, plus whole classes of filler words, and words themselves are on average substantially shorter. If you listen to or read a hyper-literal transliteration of Chinese speech into English (you can find fun videos of this on Chinese social media), it even resembles "caveman speech" for those reasons. If you look at…

Chinese omits articles, verbs aren't conjugated, and individual characters carry more meaning than English letters, but other than those differences I don't have the impression that Chinese communication is inherently more concise. Some forms of official speech are wordy. Writing is denser, but the amount of information conveyed through speech is about the same. There are jokes about ambiguous words or phrases in bot…

That's a really interesting point about Ancient Chinese and other ancient scripts. I'd love to learn more about that.

I'm also more curious about tokenizers for LLMs than I've ever been before, both for Chinese and English. I feel like to understand I'll need to look at some concrete examples, since sometimes tokenization can be per word or per character or sometimes chunks that are in between.

Re: Caveman: Why use many token when few token do trick

#305

Post by lelanthran »

teekert wrote: post #10

Idk I try talk like cavemen to claude. Claude seems answer less good. We have more misunderstandings. Feel like sometimes need more words in total to explain previous instructions. Also less context is more damage if typo. Who agrees? Could be just feeling I have. I often ad fluff. Feels like better result from LLM. Me think LLM also get less thinking and less info from own previous replies if talk like caveman.

I once (when ChatGPT first came out) launched into a conversation with ChatGPT using nothing but s-expressions. Didn't bother with a preamble, nor an explanation, just structured my prompt into a tree, forced said tree into an s-expression and hit enter.

I was very surprised to see that the response was in s-expressions too. It was incoherent, but the parens balanced at least.

Just tried it now and it doesn't seem to do that anymore.

Re: Caveman: Why use many token when few token do trick

#306

Post by alfanick »

999900000999 wrote: post #295
alfanick wrote: post #283

Either this already exists, or someone is going to implement that (should I implement that?): - assumption LLM can input/output in any useful language, - human languages are not exactly optimal away to talk with LLM, - internally LLMs keep knowledge as whole bunch of connections with some weights and multiple layers, - they need to decode human-language input into tokens, then into something that is easy to digest by…

My theory was that someone should write a specific LLM language, and then spend a whole lot of money to train models using that. A few times other commenters here have pointed out that that would be really difficult . But I think you're onto something, human languages just aren't optimal here. But to actually see this product to conclusion you'd probably need 60 to 100 million. You would have to completely invent a n…

I'm currently downloading Ollama and going to write a simple proof-of-concept with Qwen as local "frontend", talking to OpenAI GPT as "backend". I think the idea is sound, but indeed needs retraining of GPT (hmm like training tiny local LLM in synchronization of a big remote LLM). It might be not bad business venture in the end.

I don't think humans should be involved in developing this AI-AI language, just giving some guidance, but let two agents collaborate to invent the language, and just gratify/punish them with RL methods.

OpenAI looking at you, got an email some days ago "you're not using OpenAI API that much recently, what changed?"

Re: Caveman: Why use many token when few token do trick

#307

Post by sumeno »

sgbeal wrote: post #292
JBrussee-2 wrote: post #208

Author here. A few people are arguing against a stronger claim than the repo is meant to make. As well, this was very much intended to be a joke and not research level commentary. This skill is not intended to reduce hidden reasoning / thinking tokens. Anthropic’s own docs suggest more thinking budget can improve performance, so I would not claim otherwise. What it targets is the visible completion: less preamble, le…

> There is actual research suggesting concise prompting can reduce response length substantially without always wrecking quality, Anecdote: i discussed that with an LLM once and it explained to me that LLMs tend to respond to terse questions with terse answers because that's what humans (i.e. their training data) tend to do. Similarly, it explained to me that polite requests tend to lead to LLM responses with _more_…

LLMs don't understand what they are doing, they can't explain it to you, it's just creating a reasonable sounding response

Re: Caveman: Why use many token when few token do trick

#308

Post by krackers »

dTal wrote: post #302
krackers wrote: post #282

Earlier quoted context omitted.

>They aren't stenographically hiding useful computation state in words like "the" and "and". When producing a token the model doesn't just emit the final token but you also have the entire hidden states from previous attention blocks. These hidden states are mixed into the attention block of future tokens (so even though LLMs are autoregressive where a token attends to previous tokens, in terms of a computational gra…

Huh okay, there was a major gap in my mental model. Thanks for helping to clear it up.

Well to be fair the fact that they "can" doesn't mean models necessarily do it. You'd need some interp research to see if they actually do meaningfully "do other computations" when processing low perplexity tokens. But the fact that by the computational graph the architecture should be capable of it, means that _not_ doing this is leaving loss on the table, so hopefully optimizer would force it to learn to so.

Re: Caveman: Why use many token when few token do trick

#310

Post by kristopolous »

WarmWash wrote: post #213
teekert wrote: post #10

Idk I try talk like cavemen to claude. Claude seems answer less good. We have more misunderstandings. Feel like sometimes need more words in total to explain previous instructions. Also less context is more damage if typo. Who agrees? Could be just feeling I have. I often ad fluff. Feels like better result from LLM. Me think LLM also get less thinking and less info from own previous replies if talk like caveman.

In the regular people forums (twitter, reddit), you see endless complaints about LLMs being stupid and useless. But you also catch a glimpse of how the author of the complaint communicates in general... "im trying to get the ai to help with the work i am doing to give me good advice for a nice path to heloing out and anytim i askin it for help with doing this it's total trash i dunt kno what to do anymore with this d…

The realization is LLMs are computer programs. You orchestrate them like any other program and you get results.

Everyone's interfaces, concept and desires are different so the performance is wildly varied

This is similar to frameworks: they were either godsends or curses depending on how you thought and what you were doing ..

Post reply on HN