Executing programs inside transformers with exponentially faster inference
1–10 of 139 posts
Re: Executing programs inside transformers with exponentially faster inference
#2This seems a really interesting path for interpretability, specially if a big chunk of a model's behavior occurs pseudo-symbolically. This is an idea I had thought about, integrating tools into the main computation path of a model, but I never imagined that it could be done efficiently with just a vanilla transformer.
Truly, attention is all you need (I guess).
Re: Executing programs inside transformers with exponentially faster inference
#3one of the most interesting pieces I've read recently. Not sure I agree with all the statements there (e.g. without execution the system has no comprehension) - but extremely cool
Re: Executing programs inside transformers with exponentially faster inference
#4It makes sense that a next token predictor could execute assembly code. This is fascinating work, especially with the memory implementation.
Re: Executing programs inside transformers with exponentially faster inference
#5I'd like to see this combined with reinforcement learning to optimize models to think computationally. Generating ideas with hypothetical results and then running them in the same thought. Their solution sounded like a lot of tokens though.
Re: Executing programs inside transformers with exponentially faster inference
#6Is this genius? Or just a new binary executable format? Can't tell.
Re: Executing programs inside transformers with exponentially faster inference
#7This is brilliant, game changing level.
Hey, give it also access to the dump of its weights and way to propose updates so it can see and tinker its brain directly.
Re: Executing programs inside transformers with exponentially faster inference
#8what!
Re: Executing programs inside transformers with exponentially faster inference
#9This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message.
> This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the same transformer.
What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop through this computation? Do you do so?
Why is it good that it's "inside" the model? Just making it more elegant and nice? The tool was already "inside" the overall hybrid system. What's the actual problem?
Re: Executing programs inside transformers with exponentially faster inference
#10big question is how efficient is this compare to executing assembly on CPU