Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing programs inside transformers with exponentially faster inference

percepta.ai

Post reply on HN
Print view View original

71–80 of 139 posts

Re: Executing programs inside transformers with exponentially faster inference

#71

Post by D-Machine »

radarsat1 wrote: post #28
bonoboTP wrote: post #9

This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…

> Is it speed? > Is it that you can backprop through this computation? Do you do so? With respect, I feel that you may not have read the article. > Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate…

Did you read the post you are responding to? It says:

> What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop through this computation? Do you do so?

The correct parsing of this is: "What's the benefit? [...] Is it [the benefit] that you can backprop through this computation? Do you do so?"

There are no details about training nor the (almost-certainly necessarily novel) loss function that would be needed to handle partial / imperfect outputs here, so it is extremely hard to believe any kind of gradient-based training procedure was used to determine / set weight values here.

Re: Executing programs inside transformers with exponentially faster inference

#72

Post by D-Machine »

dnautics wrote: post #69

the paper is burying the lede here (i think?) > The key technical unlock is to restrict lookup heads to head dimension 2, which enables a decoding path where the dominant retrieval/update operations can be computed in log time in the sequence length (for this structured executor regime), rather than by a full prefix-sized attention sweep. edit: i understand how hullkv works now. very clever. I dont understand why thi…

> i have a pretty good understanding of how transformers work but this did not make sense to me. also i dont understand why this strategy is applicable only to "code tokens"

Yes, there is a monstrous lack of detail here and you should be skeptical about most of the article claims. The language is also IMO non-standard (serious people don't talk about self-attention as lookup tables anymore, that was never a good analogy in the first place) and no good work would just use language to express this, there would also be a simple equation showing the typical scaled dot-product attention formula, and then e.g. some dimension notation/details indicating which matrix (or inserted projection matrix) got some dimension of two somewhere, otherwise, the claims are inscrutable (EDIT: see edit below).

There are also no training details or loss function details, both of which would be necessary (and almost certainly highly novel) to make this kind of thing end-to-end trainable, which is another red flag.

EDIT: The key line seems to be around:

    gate, val = ff_in(x).chunk(2, dim=-1)
and related code, plus the lines "Notice: d_model = 36 with n_heads = 18 gives exactly 2D per head" but, again, this is very unclear and non-standard.

Re: Executing programs inside transformers with exponentially faster inference

#73

Post by vuciuc »

mobilejdral wrote: post #31
MattPalmer1086 wrote: post #11

Interesting... But why? What is the benefit, other than increasing our understanding of model architectures? Our brains can also simulate turing machines, slowly. We automated that with computers that are faster and more reliable. So why not allow a model to use external much faster and reliable tools, just as we do?

I spent the entire time reading it pondering the same thing. 1. The article presents that calling out to a tool like python is "expensive" because of the overhead of forking a process, loading up the python env etc, but why not just eliminate that overhead and embed WebAssembly so this "tool call" is near zero? This feels very similar to the discussion in the 90's around the overhead of threads v.s. processes or kern…

> "A system that cannot compute cannot truly internalize what computation is."

The way this is formulated, almost sounds like they think that giving llms this ability will bring them closer to having experiences of computation or smth? Weird?

Re: Executing programs inside transformers with exponentially faster inference

#74

Post by soulofmischief »

famouswaffles wrote: post #65
soulofmischief wrote: post #55

Earlier quoted context omitted.

> Not really sure what this obsession with calling things you don't like AI generated is but it's poor form Admonishing someone for correctly identifying AI-written or AI-edited blog posts is poor form, friend. It is without a doubt written by an LLM. All of the telltale signs are there. I work with these tools 8-20 hours a day and after a while the verbiage and grammatical structures stick out like a sore thumb. Get…

I don't care how sure you are. Honestly, it's irrelevant. 99% of the time, it's a more pleasant and productive conversation for everyone involved if you just focus on issues you had with the text itself than any nebulous AI involvement. From my point of view, all you've done is said a lot of nonsense and fabricated a convoluted explanation for why you think the text is bad. I'm fine on my horse thanks.

People can no longer freely point out when the fact that a piece of work is automated and the lack of meat are red flags as to the veracity of the content, but your antagonistic metacommentary for other people pointing out factual information is welcome discourse?

You claimed "this obsession with calling things you don't like AI generated" is "poor form", attacking the parent commenter by claiming they are lying about the nature of the content. However, multiple people have pointed out the clear signs which you missed, and the consensus is that you were wrong. Now you suddenly don't care about this point, and have introduced a new argument instead.

"From my point of view, all you've done is said a lot of nonsense and fabricated a convoluted explanation for why you think the text is bad"

What a bad-faith response. Categorically dismissive, vague, antagonistic and ultimately failing to critically engage with anything I said.

Re: Executing programs inside transformers with exponentially faster inference

#75

Post by teiferer »

Why would that be desirable?

If we take the human brain as an example, it's pretty bad at computation. Multiply two 10-digit numbers takes forever, despite the enormous size of its neural network. It's not the right tool for the job - a few deterministic logic gates could do that much more efficiently. That same circuit can't do much else, but multiplying, oh boy, it's good at that! Why do we think that artificial neural nets would be the right tool for that job? What's wrong with letting the LLM reach out to an ALU to do the calculation, just like a human would do? It's surely going to be quicker and require less energy.

Re: Executing programs inside transformers with exponentially faster inference

#76

Post by hashmap »

this is neat but to me seems like the circuitous path to just skipping autoregression, whereas the direct path is to just not do autoregression. get your answers from the one forward pass, and instead of backprop just do lookups and updates as the same operation.

Re: Executing programs inside transformers with exponentially faster inference

#77

Post by famouswaffles »

soulofmischief wrote: post #74
famouswaffles wrote: post #65

Earlier quoted context omitted.

I don't care how sure you are. Honestly, it's irrelevant. 99% of the time, it's a more pleasant and productive conversation for everyone involved if you just focus on issues you had with the text itself than any nebulous AI involvement. From my point of view, all you've done is said a lot of nonsense and fabricated a convoluted explanation for why you think the text is bad. I'm fine on my horse thanks.

People can no longer freely point out when the fact that a piece of work is automated and the lack of meat are red flags as to the veracity of the content, but your antagonistic metacommentary for other people pointing out factual information is welcome discourse? You claimed "this obsession with calling things you don't like AI generated" is "poor form", attacking the parent commenter by claiming they are lying abou…

Whether a piece of work is automated and 'lacks meat' is ultimately not something you can know for sure as a reader. Articles like this existed plenty Pre-AI and will exist plenty post-AI, involvement or not, so yeah pretty pointless to focus on that. It adds nothing and all we have to go is your own surety, which is fallible. If you can't recognize that then there's not much to say.

I didn't miss anything. I never cared about it one way or another. What clear signs have people pointed out ? This is the problem. It's apparently so obvious yet even the original commenter admits "It's things humans do too". What is clear about that ?

Re: Executing programs inside transformers with exponentially faster inference

#79

Post by refulgentis »

This sounds so cool but I can’t tell if it’s a practical joke, even after sitting on it for 2-3 hours. Key points where I lose understanding/trust are when a WASM interpreter suddenly appears in the model, and when we’re representing code in weights.

It is unclear to me how this WASM interpreter is / could be deterministic.

Re: Executing programs inside transformers with exponentially faster inference

#80

Post by soerxpso »

teiferer wrote: post #75

Why would that be desirable? If we take the human brain as an example, it's pretty bad at computation. Multiply two 10-digit numbers takes forever, despite the enormous size of its neural network. It's not the right tool for the job - a few deterministic logic gates could do that much more efficiently. That same circuit can't do much else, but multiplying, oh boy, it's good at that! Why do we think that artificial ne…

The embedded programs can be connected to the other weights during training, in whatever way the training process finds useful. It doesn't just have to be arithmetic calculation. You can put any hard-coded algorithm in there, make the weights for that algorithm static, and let the training process figure out how to connect the other trillion weights to it.
Post reply on HN