Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing programs inside transformers with exponentially faster inference

percepta.ai

Post reply on HN
Print view View original

121–130 of 139 posts

Re: Executing programs inside transformers with exponentially faster inference

#121

Post by Oranguru »

bsenftner wrote: post #34
bonoboTP wrote: post #9

This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…

Well, for one, by eliminating external tool calling, the model gains an amount of security. This occurs because the tools being called by an LLM can be corrupted, and in this scenario corrupted tools would not be called.

Prompt injection is still a possibility, so while it improves the security posture, not by much.

Re: Executing programs inside transformers with exponentially faster inference

#123

Post by krackers »

deviation wrote: post #15

I really liked the article, but food for thought: is a transformer that offloads computation to python really that different from Python code being read and then executed by a compiler? Both examples are of a system we created to abstract most of the hard work. I think a more important concept here is that the term "AI" has a lot of built-in assumptions, one of which being that it is (or will be) super intelligent, a…

There's one difference that if a program is run as tool call, the internal states and control flow are not visible to the LLM. You can imagine this being useful for "debugging" in a meta-sense, the same way humans can use debuggers to figure out where something went awry it might be useful for the LLM to "simulate" something and have access to the execution trace.

Of course you can also just simulate this by peppering your code with print statements, so maybe it's not that useful in the end after all.

Re: Executing programs inside transformers with exponentially faster inference

#124

Post by ACCount37 »

YeGoblynQueenne wrote: post #63
ACCount37 wrote: post #38

This seems like it has some potential, but is pretty much useless as it is. Shame there are no weights released - let alone the "compiler" tool they used to actually synthesize computational primitives into model weights. It seems like a "small model" system that's amenable to low budget experiments, and I would love to see what this approach can be pushed towards. I disagree with the core premise, it's basically the…

What's "the old neurosymbolic garbage"?

Basically, a holdover from the days of symbolic AI, from back when neural network ML wasn't the dominant AI paradigm.

Some people in the "symbolic AI" camp didn't take the loss well, so they pivoted towards "ML is not real AI and it needs a symbolic component to be a real AI", which is: the neurosymbolic garbage.

This work isn't exactly that, and I do think it can amount to something useful, but the justification for it reeks of something similar.

Re: Executing programs inside transformers with exponentially faster inference

#125

Post by YeGoblynQueenne »

D-Machine wrote: post #110
gavinray wrote: post #106

Earlier quoted context omitted.

The article states they trained a WASM interpreter and programs are represented as WASM bytecode

Nope, they encoded or compiled in a simple VM / WASM interpreter to the transformer weights, there is no training. You'd be forgiven for this misreading, as they deliberately mislead early on that their model is (in principle) trainable, but later admit that their actual model is not actually differentiable, but that a differentiable approximation "should" still work (despite no info about what loss function or train…

Thanks, but where do they say that? I can only find this instance of "different" (as in "differentiable") in their article:

Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.

Re: Executing programs inside transformers with exponentially faster inference

#126

Post by YeGoblynQueenne »

MadnessASAP wrote: post #89
YeGoblynQueenne wrote: post #59

So, what I'm trying to understand, and I can't find any clear information about that in the article, is how they "compiled" e.g. the Sudoku solver into a Transformer's weights. Did they do it manually? Say, they took the source of a hand-coded Sudoku solver and put it through their code-to-weight compiler, and thus compiled the code to the Transformer weights? Or did they go the Good, Old-Fashioned, Deep Learning way…

My interpretation is that they built a simple virtual machine directly into the weights, then compiled a WASM runtime for that machine, then compiled the solver to that runtime.

That's more or less what I got, also, but it's hard to tell. What a very annoying article, in its vagueness.

Re: Executing programs inside transformers with exponentially faster inference

#127

Post by YeGoblynQueenne »

D-Machine wrote: post #105
dwa3592 wrote: post #67

Earlier quoted context omitted.

I assumed that they had to train, otherwise how else would they get "inside" a transformer. I also feel a bit of bad smell from the article. Sounding revolutionary with no details or clear explanation.

There is no training in the usual sense of the term, i.e. no gradient descent, no differentiable loss function. They use deceptive language early on to make it sound this way, but near the end make it clear their model as is isn't actually differentiable, and in theory might still work if made differentiable . But they don't actually know. But IMO this is BS because I don't know how one would get or generate training…

Yeah, a "100% correct" Sudoku solver fully trained by gradient descent from examples? That sure would be something entirely new.

To answer dwa3592, it's always possible to set the weights of a neural net by hand, albeit extremely fiddly and normally only done "on paper". This is e.g. how the Turing-completeness of RNNs was shown back in the '90s:

On the computational power of neural nets

https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf

Re: Executing programs inside transformers with exponentially faster inference

#128

Post by YeGoblynQueenne »

ACCount37 wrote: post #124
YeGoblynQueenne wrote: post #63

Earlier quoted context omitted.

What's "the old neurosymbolic garbage"?

Basically, a holdover from the days of symbolic AI, from back when neural network ML wasn't the dominant AI paradigm. Some people in the "symbolic AI" camp didn't take the loss well, so they pivoted towards "ML is not real AI and it needs a symbolic component to be a real AI", which is: the neurosymbolic garbage. This work isn't exactly that, and I do think it can amount to something useful, but the justification for…

Full disclosure: all my published work is on symbolic machine learning (a.k.a. Inductive Logic Programming) :O

I think you're confusing various different things as "neurosymbolic AI". There is a NeSy symposium and I happen to have met many of the people there, and they are not GOFAI ideologues, rather they recognise the obvious limitations of neural nets (i.e. they're crap at deduction, though great at induction) and they look for ways to address them. Most of that crowd also has a predominantly statistical ML/ neural nets background, with symbolic AI as an afterthought.

I don't think I've ever heard anyone say that "ML is not real AI" and I mainly move in symbolic AI circles. I would check my sources, if I were you.

Anwyay, honestly, this is 2026, there is no sensible reason to be polarised about symbolic vs. statistical AI (or whatever distinction anyone wants to make). An analogy I like to make is as follows: a jetliner is a flying machine, a helicopter is a flying machine. We can use both for their advantages and disadvantages, but a flying machine is something too useful to give up on any one kind for ideological reasons. The practical benefits overwhelmingly make up for any ideological concerns (e.g. "jets bad" or "propellers bad").

And just to be clear, symbolic AI is still in rude health: automated theorem proving, planning and scheduling, program verification and model checking, constraint satisfaction, discrete optimisation, SAT solving, all those are fields where symbolic approaches are dominant, and where neural nets have not made significant inroads in many decades; nor are they likely to, not any more than symbolic approaches are likely to make any inroads in e.g. machine vision, or speech recognition. And that's just fine: lots of tools, lots of problems solved.

Re: Executing programs inside transformers with exponentially faster inference

#129

Post by ACCount37 »

YeGoblynQueenne wrote: post #128
ACCount37 wrote: post #124

Earlier quoted context omitted.

Basically, a holdover from the days of symbolic AI, from back when neural network ML wasn't the dominant AI paradigm. Some people in the "symbolic AI" camp didn't take the loss well, so they pivoted towards "ML is not real AI and it needs a symbolic component to be a real AI", which is: the neurosymbolic garbage. This work isn't exactly that, and I do think it can amount to something useful, but the justification for…

Full disclosure: all my published work is on symbolic machine learning (a.k.a. Inductive Logic Programming) :O I think you're confusing various different things as "neurosymbolic AI". There is a NeSy symposium and I happen to have met many of the people there, and they are not GOFAI ideologues, rather they recognise the obvious limitations of neural nets (i.e. they're crap at deduction, though great at induction) and…

I don't think symbolic approaches are completely useless. It's just that they're solving yesterday's problems 1.12% better. While ML is cracking open entirely new fields - and might go all the way to AGI, the way it's going now.

One is near the end of its potential while another is only picking up steam.

In many ways, the space ML dominates now is the space of "all the things symbolic approaches suck ass at". Which is a very wide space with many desirable things in it.

Re: Executing programs inside transformers with exponentially faster inference

#130

Post by alasr »

btown wrote: post #17

This seems way cooler than just computation (which is easy to hand off to a tool, and arguably more predictable that way). The broader point here is that you can have your model switch dynamically to/from a kind of attention that scales with the log of the token count, by only exploring the convex hull in a 2D space. A less capable version of attention, to be sure, but one capable of tracing a program’s execution wit…

Just a small nitpik with the following part of your post.

> As the paper suggests: ...

This is not a paper; it's just a blog post which is mentioned in this post's URL (i.e. .../blog/can-llms-be-computers).

Also it, at the end of the article, has a job advertisement (nothing unusual for a blog post):

    "We are building these systems now, and we are hiring. If you want to work on problems at ..., join us."
Post reply on HN