very cool idea. But, time savings are not true for every tool call, and it's not clear to me yet whether this is batch-able; also, intuitively, for most of the models that run on GPU, you'd still want to offload tool exec part to CPU since it's much cheaper...
Executing programs inside transformers with exponentially faster inference
21–30 of 139 posts
Re: Executing programs inside transformers with exponentially faster inference
#22Interesting... But why? What is the benefit, other than increasing our understanding of model architectures? Our brains can also simulate turing machines, slowly. We automated that with computers that are faster and more reliable. So why not allow a model to use external much faster and reliable tools, just as we do?
Why must models be analogous to humans using tools? Or to take the analogy route further wouldn't it be better if humans had calculators built into their brains, provided they are determisitic and reduce latency
Re: Executing programs inside transformers with exponentially faster inference
#23This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…
>This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. Not really sure what this obsession with calling things you don't like AI generated is but it's poor form. If you have something to say about the text then say it. Otherwise leave baseless accusations out of it. >What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop…
This is a nice case study of the downside of creating explicit policies of "no AI comments" without a technical method of enforcing it. I am sure the hacker news comment quality will suffer almost as much from an escalating culture of accusation and paranoia that it will from LLM comment themselves.
Re: Executing programs inside transformers with exponentially faster inference
#24Earlier quoted context omitted.
>"This shows the downside of using AI to write up your project." I just find phrases like this a bit obnoxious at times. >You would not have had a problem with calling out a badly composed rambling article 5 years ago. Then why not just say that? It's rambling bla bla bla. What's so hard about that? Why invent a reason for issues, as if rambling articles didn't get written 5 years ago. Like No, being written by an LL…
It's very hard to discuss this. To some people it's obvious, to some it isn't. To me, every single paragraphs is obvious fluff AI writing. One problem with it is the repetitiveness and the schmoozing salesman feel. The other is the lack of benchmarks and stuff. It's both. The two are connected because the AI has to lean in to its bullshitter persona when it's not given enough raw material to write up something strong…
For what it’s worth, I agree with you; the article is LLM written although not with the usual gotchas, so they’re more subtle.
The subtle ones like this I don’t mind too much, as long as they get the content correct, which in this case leaves quite a bit to be desired.
I’m also noticing that some people around me appear to just be oblivious to some LLM signals that bother me a lot, so people consume media differently.
I absolutely do believe that AI generated content needs to be called out, although at this point it’s safe to say that pretty much all online content is LLM written.
Re: Executing programs inside transformers with exponentially faster inference
#25This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…
What are the AI tells? The only one I found is redundancy, but it makes sense because this is trying to be approachable to laymen.
Like, you have a great point (the benefit of this approach isn't explained), but that's a mistake humans frequently make.
Re: Executing programs inside transformers with exponentially faster inference
#26The original title is "Can LLMs be computers?"
But the right question is, should they?
Re: Executing programs inside transformers with exponentially faster inference
#27This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…
The key difference is that the model is able to write the program as it’s executing it.
Before it needs to write the code and have an external program execute it. Here it can change its mind mid execution. Kinda like what was observed in the CoT’s ah ha moment
Re: Executing programs inside transformers with exponentially faster inference
#28This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…
> Is it speed?
> Is it that you can backprop through this computation? Do you do so?
With respect, I feel that you may not have read the article.
> Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.
and,
> By storing points across nested convex hulls, this yields a decoding cost of O(k+log n).
and,
> Regardless of their eventual capability ceiling, they already suggest a powerful systems primitive for speeding up larger models.
So yes, and yes.
> Where are the benchmarks?
Not clear what they should benchmark it against. They do compare speed to a normal KV Cache. As for performance.. if it's actually executing a Sudoku solver with a 100% success rate, it seems pretty trivial to find any model doing Personally I think it would be really interesting to see if this method can be combined with a normal model MoE-style. It is likely possible, the router module should pick up quite quickly that it predicts the right tokens for some subset of problems deterministically. I like the idea of embed all sorts of general solvers directly into the model, like a prolog solver for example. In fact it never would have occurred to me to just go straight for WASM, pretty interesting choice to directly embed a VM. But it makes me wonder what "smaller" interpreters could be useful in this context.
Re: Executing programs inside transformers with exponentially faster inference
#29This is really important work.
Re: Executing programs inside transformers with exponentially faster inference
#30This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…
>This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. Not really sure what this obsession with calling things you don't like AI generated is but it's poor form. If you have something to say about the text then say it. Otherwise leave baseless accusations out of it. >What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop…
Agreeing first that it is genuinely interesting, let me make a constructive comment on the text: Early on, there are too many small paragraphs that don't on their own make a cogent argument. That important but easily overlooked structural work is pushed back to the reader. I felt rewarded in pushing past that though. Bravo.