Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing programs inside transformers with exponentially faster inference

percepta.ai

Post reply on HN
Print view View original

91–100 of 139 posts

Re: Executing programs inside transformers with exponentially faster inference

#91

Post by jadbox »

mobilejdral wrote: post #31
MattPalmer1086 wrote: post #11

Interesting... But why? What is the benefit, other than increasing our understanding of model architectures? Our brains can also simulate turing machines, slowly. We automated that with computers that are faster and more reliable. So why not allow a model to use external much faster and reliable tools, just as we do?

I spent the entire time reading it pondering the same thing. 1. The article presents that calling out to a tool like python is "expensive" because of the overhead of forking a process, loading up the python env etc, but why not just eliminate that overhead and embed WebAssembly so this "tool call" is near zero? This feels very similar to the discussion in the 90's around the overhead of threads v.s. processes or kern…

Maybe this could be used as an optimizing profiler in order to inform the compiler on novel methods for improving hot sections of code?

Re: Executing programs inside transformers with exponentially faster inference

#92

Post by yorwba »

derangedHorse wrote: post #85

I initially agreed with a lot of the sentiment that asks "why," but have reframed my opinion. Instead of seeing this as a way to run programs via inference, I'm now seeing this as a way to bootstrap training. Think about the task of classification. If I have an expert system that classifies correctly 80% of the time, now I can embed it into a model and train the model to try to raise the success rate. The lower we ca…

The approach here is very bad for training though, because unlike softmax attention, average-hard attention is not differentiable with respect to the keys and queries, and if you try to fix that e.g. with straight-through estimation, the backward pass cannot be sped up in the same way as the forward pass.

Re: Executing programs inside transformers with exponentially faster inference

#94

Post by soulofmischief »

famouswaffles wrote: post #77
soulofmischief wrote: post #74

Earlier quoted context omitted.

People can no longer freely point out when the fact that a piece of work is automated and the lack of meat are red flags as to the veracity of the content, but your antagonistic metacommentary for other people pointing out factual information is welcome discourse? You claimed "this obsession with calling things you don't like AI generated" is "poor form", attacking the parent commenter by claiming they are lying abou…

Whether a piece of work is automated and 'lacks meat' is ultimately not something you can know for sure as a reader. Articles like this existed plenty Pre-AI and will exist plenty post-AI, involvement or not, so yeah pretty pointless to focus on that. It adds nothing and all we have to go is your own surety, which is fallible. If you can't recognize that then there's not much to say. I didn't miss anything. I never c…

Your inability to recognize the clear imprint of current-generation language models on this article doesn't mean they're not present.

All knowledge is ultimately fallible, but ignoring or not being able to appreciate the high statistical likelihood of this article being LLM edited/generated doesn't change reality.

You're asking me to share my expertise with you so that you can understand, but your antagonistic overtones make it not feel worth the time and effort. Other readers have also pointed out that it has characteristic idiosyncrasies. Feel free to look into it yourself, but it would also be wise to learn to defer these kinds of attacks until you have all the information.

Re: Executing programs inside transformers with exponentially faster inference

#95

Post by refulgentis »

derangedHorse wrote: post #85

I initially agreed with a lot of the sentiment that asks "why," but have reframed my opinion. Instead of seeing this as a way to run programs via inference, I'm now seeing this as a way to bootstrap training. Think about the task of classification. If I have an expert system that classifies correctly 80% of the time, now I can embed it into a model and train the model to try to raise the success rate. The lower we ca…

Training is ruled out (see peer comment), however you may find this fascinating, somewhat rhymes: https://arxiv.org/abs/2603.10055

Re: Executing programs inside transformers with exponentially faster inference

#96

Post by yorwba »

D-Machine wrote: post #72
dnautics wrote: post #69

the paper is burying the lede here (i think?) > The key technical unlock is to restrict lookup heads to head dimension 2, which enables a decoding path where the dominant retrieval/update operations can be computed in log time in the sequence length (for this structured executor regime), rather than by a full prefix-sized attention sweep. edit: i understand how hullkv works now. very clever. I dont understand why thi…

> i have a pretty good understanding of how transformers work but this did not make sense to me. also i dont understand why this strategy is applicable only to "code tokens" Yes, there is a monstrous lack of detail here and you should be skeptical about most of the article claims. The language is also IMO non-standard (serious people don't talk about self-attention as lookup tables anymore, that was never a good anal…

Treating attention as a lookup operation is popular among computational complexity theorists (e.g. https://arxiv.org/abs/2310.03817 ) because it's easier to work with when you're explicitly constructing a transformer to perform a particular computation, just to demonstrate that tranformers can, in theory, perform it. That's also why there are no training details: the weights are computed directly and not trained.

Re: Executing programs inside transformers with exponentially faster inference

#97

Post by radarsat1 »

D-Machine wrote: post #71
radarsat1 wrote: post #28

Earlier quoted context omitted.

> Is it speed? > Is it that you can backprop through this computation? Do you do so? With respect, I feel that you may not have read the article. > Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate…

Did you read the post you are responding to? It says: > What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop through this computation? Do you do so? The correct parsing of this is: "What's the benefit? [...] Is it [the benefit] that you can backprop through this computation? Do you do so?" There are no details about training nor the (almost-certainly necessarily novel) loss function…

> There are no details about training

my understanding was that they are not training at all, which would explain that. they are compiling an interpreter down to a VM that has the shape of a transformer.

ie they are calculating the transformer weights needed to execute the operations of the machine they are generating code for.

Re: Executing programs inside transformers with exponentially faster inference

#98

Post by D-Machine »

dnautics wrote: post #90
D-Machine wrote: post #72

Earlier quoted context omitted.

> i have a pretty good understanding of how transformers work but this did not make sense to me. also i dont understand why this strategy is applicable only to "code tokens" Yes, there is a monstrous lack of detail here and you should be skeptical about most of the article claims. The language is also IMO non-standard (serious people don't talk about self-attention as lookup tables anymore, that was never a good anal…

> lookup tables anymore, that was never a good analogy in the first place good analogy otherwise, wasn't hash tables the motivation for the kv tables?

Well, one can never be sure what the real motivation for a lot of DL advances, as most papers are post-hoc obscurantism / hand-waving or even just outright nonsense (see: internal covariate shift explanations for batch norm, which arguably couldn't be more wrong https://arxiv.org/pdf/1805.11604).

When you really get into this stuff, you tend to see the real motivations as either e.g. kernel smoothing (see comments / discussion at https://news.ycombinator.com/item?id=46357675#46359160) or as encoding correlations / feature similarities / multiplicative interactions (see e.g. broad discussion at https://news.ycombinator.com/item?id=46523887). IMO most insights in LLM architectures and layers tends to come from intuitions about projections, manifolds, dimensionality, smoothing/regularization, overparameterization, matrix conditioning, manifold curvature and etc.

There are almost zero useful understandings or insights to be gained from the lookup-table analogy, and most statistical explanations in papers are also post-hoc and require assumptions (convergence rates, infinite layers, etc) that are never shown to clearly hold for actual models that people use. Obviously these AI models work very well for a lot of tasks, but our understanding of why they do is incredibly poor and simplistic, for the most part.

Of course, this is just IMO, and you can see some people in the linked threads do seem to find the lookup table analogies useful. I doubt such people have spent much time building novel architectures, experimenting with different layers, or training such models.

Re: Executing programs inside transformers with exponentially faster inference

#99

Post by D-Machine »

yorwba wrote: post #96
D-Machine wrote: post #72

Earlier quoted context omitted.

> i have a pretty good understanding of how transformers work but this did not make sense to me. also i dont understand why this strategy is applicable only to "code tokens" Yes, there is a monstrous lack of detail here and you should be skeptical about most of the article claims. The language is also IMO non-standard (serious people don't talk about self-attention as lookup tables anymore, that was never a good anal…

Treating attention as a lookup operation is popular among computational complexity theorists (e.g. https://arxiv.org/abs/2310.03817 ) because it's easier to work with when you're explicitly constructing a transformer to perform a particular computation, just to demonstrate that tranformers can, in theory, perform it. That's also why there are no training details: the weights are computed directly and not trained.

This is a good link and important (albeit niche) qualification.

It is hard to square with the article's claims about differentiability and otherwise lack of clarity / obscurantism about what they are really doing here (they really are just compiling / encoding a simple computer / VM into a slightly-modified transformer, which, while cool, is really not what they make it sound like at all).

Re: Executing programs inside transformers with exponentially faster inference

#100

Post by D-Machine »

radarsat1 wrote: post #97
D-Machine wrote: post #71

Earlier quoted context omitted.

Did you read the post you are responding to? It says: > What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop through this computation? Do you do so? The correct parsing of this is: "What's the benefit? [...] Is it [the benefit] that you can backprop through this computation? Do you do so?" There are no details about training nor the (almost-certainly necessarily novel) loss function…

> There are no details about training my understanding was that they are not training at all, which would explain that. they are compiling an interpreter down to a VM that has the shape of a transformer. ie they are calculating the transformer weights needed to execute the operations of the machine they are generating code for.

This is my interpretation as well.

EDIT: Actually, they do make this clear(ish) at the very end of the article, technically. But there is a huge amount of vagueness and IMO outright misleading / deliberately deceptive stuff early on (e.g. about potential differentiability of their approach, even though they admit later they aren't sure if the differentiable approach can actually work for what they are doing). It is hard to tell what they are actually claiming unless you read this autistically / like a lawyer, but that's likely due to a lack of human editing and too much AI assistance.

Post reply on HN