big question is how efficient is this compare to executing assembly on CPU
My guess would be in the ballpark of about 10000 times less efficient
Executing programs inside transformers with exponentially faster inference
61–70 of 139 posts
Re: Executing programs inside transformers with exponentially faster inference
#62if you understood the article, please correct my understanding - they created a new training dataset which also has computation solving step by step (multiplying two numbers or playing sudoku) and then trained a transformer on it- as a result, the model performs the computation(multiplying two numbers) "inside" itself instead of calling calculator (or python)? ++ And they also figured out how to make attention faster…
I can't see anything about "training a transformer". I'm trying to understand if e.g. the Sudoku solver was learned from examples (in which case, what examples?) or whether it was manually coded and then "compiled" into weights.
Re: Executing programs inside transformers with exponentially faster inference
#63This seems like it has some potential, but is pretty much useless as it is. Shame there are no weights released - let alone the "compiler" tool they used to actually synthesize computational primitives into model weights. It seems like a "small model" system that's amenable to low budget experiments, and I would love to see what this approach can be pushed towards. I disagree with the core premise, it's basically the…
What's "the old neurosymbolic garbage"?
Re: Executing programs inside transformers with exponentially faster inference
#64This has a lot of potential. Especially if the compiled "code" can be efficiently shared between models of the same architecture. That would easily overshadow LoRa and finetuning in general.
Re: Executing programs inside transformers with exponentially faster inference
#65Earlier quoted context omitted.
>This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. Not really sure what this obsession with calling things you don't like AI generated is but it's poor form. If you have something to say about the text then say it. Otherwise leave baseless accusations out of it. >What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop…
> Not really sure what this obsession with calling things you don't like AI generated is but it's poor form Admonishing someone for correctly identifying AI-written or AI-edited blog posts is poor form, friend. It is without a doubt written by an LLM. All of the telltale signs are there. I work with these tools 8-20 hours a day and after a while the verbiage and grammatical structures stick out like a sore thumb. Get…
I don't care how sure you are. Honestly, it's irrelevant. 99% of the time, it's a more pleasant and productive conversation for everyone involved if you just focus on issues you had with the text itself than any nebulous AI involvement.
From my point of view, all you've done is said a lot of nonsense and fabricated a convoluted explanation for why you think the text is bad. I'm fine on my horse thanks.
Re: Executing programs inside transformers with exponentially faster inference
#66Earlier quoted context omitted.
>This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. Not really sure what this obsession with calling things you don't like AI generated is but it's poor form. If you have something to say about the text then say it. Otherwise leave baseless accusations out of it. >What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop…
> Not really sure what this obsession with calling things you don't like AI generated is but it's poor form Admonishing someone for correctly identifying AI-written or AI-edited blog posts is poor form, friend. It is without a doubt written by an LLM. All of the telltale signs are there. I work with these tools 8-20 hours a day and after a while the verbiage and grammatical structures stick out like a sore thumb. Get…
The post is the perfect example of the kind of writing about AI that dupes people that don't really understand how things like LLMs actually work and are actually trained. Anyone who properly understands these things finds the complete and total lack of detail about training and the loss function (and of course real metrics / benchmarks) to be a monstrous red flag here.
Especially egregious to me is the claim "Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself". This is total weasel-language: e.g. we can propagate any weights through any transformer architecture and all sorts of other much more insane architectural designs, but that is irrelevant if you don't have a continuous and differentiable loss function that can properly weight partially-correct solutions or the likelihood / plausibility of arbitrary model outputs. You also need a clearer source of training data (or way to generate synthetic data).
So for e.g. AlphaFold, we needed to figure out a loss function that continuously approximated the energy configuration of various molecular configurations, and this is what really allowed it to actually do something. Otherwise, you are stuck with slow and expensive reinforcement-based systems.
The other tells are garbage analogies ("Humans cannot fly. Building airplanes does not change that; it only means we built a machine that flies for us"). Such analogies add nothing to understanding, and indeed distract from serious/real understanding. Only dupes and fools think you can gain any meaningful understanding of mathematics and computer science through simplistic linguistic analogies and metaphors without learning the proper actual (visuspatial, logical, etc) models and understanding. Thus, people with real and serious mathematical understanding despise such trite metaphors.
But then, since understanding something like this properly requires serious mathematical understanding, copy like that is a huge tell that the authors / company / platform puts bullshitting and sales above truth and correctness. I.e., yes, a huge yellow flag.
Re: Executing programs inside transformers with exponentially faster inference
#67if you understood the article, please correct my understanding - they created a new training dataset which also has computation solving step by step (multiplying two numbers or playing sudoku) and then trained a transformer on it- as a result, the model performs the computation(multiplying two numbers) "inside" itself instead of calling calculator (or python)? ++ And they also figured out how to make attention faster…
I can't see anything about "training a transformer". I'm trying to understand if e.g. the Sudoku solver was learned from examples (in which case, what examples?) or whether it was manually coded and then "compiled" into weights.
I assumed that they had to train, otherwise how else would they get "inside" a transformer.
I also feel a bit of bad smell from the article. Sounding revolutionary with no details or clear explanation.
Re: Executing programs inside transformers with exponentially faster inference
#68Earlier quoted context omitted.
What are the AI tells? The only one I found is redundancy, but it makes sense because this is trying to be approachable to laymen. Like, you have a great point (the benefit of this approach isn't explained), but that's a mistake humans frequently make.
I read a lot of LLM text every day, so I'm quite good at seeing the cadence, the narrative structures and the phrasing styles. It's not just "it's not just X but Y" or emdashes. I could point them out and you would say oh humans use this trope or phrasing style too, and of course that's true. It's still a tell. But it's pointless to argue about this.
[deleted]
Re: Executing programs inside transformers with exponentially faster inference
#69the paper is burying the lede here (i think?)
> The key technical unlock is to restrict lookup heads to head dimension 2, which enables a decoding path where the dominant retrieval/update operations can be computed in log time in the sequence length (for this structured executor regime), rather than by a full prefix-sized attention sweep.
edit: i understand how hullkv works now. very clever.
I dont understand why this strategy is applicable only to "code tokens"
lastly, im not sure why wasm is a good target, iirc wasm seems to be really inefficient (not so much in code but in expressivity). i wonder if that curtails the llms ability to plan higher order stuff (since its always forced to think in the small)
Re: Executing programs inside transformers with exponentially faster inference
#70if you understood the article, please correct my understanding - they created a new training dataset which also has computation solving step by step (multiplying two numbers or playing sudoku) and then trained a transformer on it- as a result, the model performs the computation(multiplying two numbers) "inside" itself instead of calling calculator (or python)? ++ And they also figured out how to make attention faster…
I can't see anything about "training a transformer". I'm trying to understand if e.g. the Sudoku solver was learned from examples (in which case, what examples?) or whether it was manually coded and then "compiled" into weights.
I would assume it was manually coded.