Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing programs inside transformers with exponentially faster inference

percepta.ai

Post reply on HN
Print view View original

81–90 of 139 posts

Re: Executing programs inside transformers with exponentially faster inference

#81

Post by pegasus »

teiferer wrote: post #75

Why would that be desirable? If we take the human brain as an example, it's pretty bad at computation. Multiply two 10-digit numbers takes forever, despite the enormous size of its neural network. It's not the right tool for the job - a few deterministic logic gates could do that much more efficiently. That same circuit can't do much else, but multiplying, oh boy, it's good at that! Why do we think that artificial ne…

> Why would that be desirable?

If we never try, we'll never know. I wouldn't be surprised if there is something to gain from a form of deterministic computation which is still integrated with the NN architecture. After all, tool calls have their own non-trivial overhead.

Re: Executing programs inside transformers with exponentially faster inference

#82

Post by teiferer »

pegasus wrote: post #81
teiferer wrote: post #75

Why would that be desirable? If we take the human brain as an example, it's pretty bad at computation. Multiply two 10-digit numbers takes forever, despite the enormous size of its neural network. It's not the right tool for the job - a few deterministic logic gates could do that much more efficiently. That same circuit can't do much else, but multiplying, oh boy, it's good at that! Why do we think that artificial ne…

> Why would that be desirable? If we never try, we'll never know. I wouldn't be surprised if there is something to gain from a form of deterministic computation which is still integrated with the NN architecture. After all, tool calls have their own non-trivial overhead.

Trying, sure. That's what hackers do.

I'm asking whether it's a desirable end state.

Re: Executing programs inside transformers with exponentially faster inference

#83

Post by D-Machine »

skybrian wrote: post #78

I couldn't tell from the article whether this works as a language model or not. Can it read and write English or is it just a weird program interpreter? If it switches between modes, how do they interact?

> In our construction, each instruction maps to only a handful of tokens (at most 5).

I don't see how this could work as an LLM given that, but the article is missing a huge amount of other crucial details too.

Re: Executing programs inside transformers with exponentially faster inference

#85

Post by derangedHorse »

I initially agreed with a lot of the sentiment that asks "why," but have reframed my opinion. Instead of seeing this as a way to run programs via inference, I'm now seeing this as a way to bootstrap training. Think about the task of classification. If I have an expert system that classifies correctly 80% of the time, now I can embed it into a model and train the model to try to raise the success rate. The lower we can make the cost of training on various tasks, the better it levels the playing field of who can compete in the AI landscape.

Re: Executing programs inside transformers with exponentially faster inference

#86

Post by OneDeuxTriSeiGo »

teiferer wrote: post #75

Why would that be desirable? If we take the human brain as an example, it's pretty bad at computation. Multiply two 10-digit numbers takes forever, despite the enormous size of its neural network. It's not the right tool for the job - a few deterministic logic gates could do that much more efficiently. That same circuit can't do much else, but multiplying, oh boy, it's good at that! Why do we think that artificial ne…

One of the big appeals of this is it gives a mechanism for "teaching" models a geometric intuition and better spacial reasoning.

Not necessarily pure number crunching but the boundary between rote algorithms and fuzzy intuition based models that humans in particular excel at.

Re: Executing programs inside transformers with exponentially faster inference

#88

Post by Hugsun »

manas96 wrote: post #43

LLMs are not deterministic per my understanding. A program always produces the same output for the same input and instructions (ignore FP accuracy for now). How is determinism achieved here?

LLMs produce a distribution of token probabilities which is then sampled. This sampling is the only random part of the system.

If you just take the most probable token every time, the system becomes fully deterministic. We don't do this as the output becomes more stiff and less creative.

Re: Executing programs inside transformers with exponentially faster inference

#89

Post by MadnessASAP »

YeGoblynQueenne wrote: post #59

So, what I'm trying to understand, and I can't find any clear information about that in the article, is how they "compiled" e.g. the Sudoku solver into a Transformer's weights. Did they do it manually? Say, they took the source of a hand-coded Sudoku solver and put it through their code-to-weight compiler, and thus compiled the code to the Transformer weights? Or did they go the Good, Old-Fashioned, Deep Learning way…

My interpretation is that they built a simple virtual machine directly into the weights, then compiled a WASM runtime for that machine, then compiled the solver to that runtime.

Re: Executing programs inside transformers with exponentially faster inference

#90

Post by dnautics »

D-Machine wrote: post #72
dnautics wrote: post #69

the paper is burying the lede here (i think?) > The key technical unlock is to restrict lookup heads to head dimension 2, which enables a decoding path where the dominant retrieval/update operations can be computed in log time in the sequence length (for this structured executor regime), rather than by a full prefix-sized attention sweep. edit: i understand how hullkv works now. very clever. I dont understand why thi…

> i have a pretty good understanding of how transformers work but this did not make sense to me. also i dont understand why this strategy is applicable only to "code tokens" Yes, there is a monstrous lack of detail here and you should be skeptical about most of the article claims. The language is also IMO non-standard (serious people don't talk about self-attention as lookup tables anymore, that was never a good anal…

> lookup tables anymore, that was never a good analogy in the first place

good analogy otherwise, wasn't hash tables the motivation for the kv tables?

Post reply on HN