Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing programs inside transformers with exponentially faster inference

percepta.ai

Post reply on HN
Print view View original

101–110 of 139 posts

Re: Executing programs inside transformers with exponentially faster inference

#101

Post by hedgehog »

mobilejdral wrote: post #31
MattPalmer1086 wrote: post #11

Interesting... But why? What is the benefit, other than increasing our understanding of model architectures? Our brains can also simulate turing machines, slowly. We automated that with computers that are faster and more reliable. So why not allow a model to use external much faster and reliable tools, just as we do?

I spent the entire time reading it pondering the same thing. 1. The article presents that calling out to a tool like python is "expensive" because of the overhead of forking a process, loading up the python env etc, but why not just eliminate that overhead and embed WebAssembly so this "tool call" is near zero? This feels very similar to the discussion in the 90's around the overhead of threads v.s. processes or kern…

I'm not sure about the rest but a significant problem with high frequency tool calling (especially in training) is that it breaks batching.

Re: Executing programs inside transformers with exponentially faster inference

#102

Post by D-Machine »

armchairhacker wrote: post #25
bonoboTP wrote: post #9

This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the sa…

What are the AI tells? The only one I found is redundancy, but it makes sense because this is trying to be approachable to laymen. Like, you have a great point (the benefit of this approach isn't explained), but that's a mistake humans frequently make.

Here is a rough list, some may be contentious individually, but the more of these appear, the more you should suspect an LLM:

Cadence and rhythm: LLMs produce sentences with an extremely low variability in the number of clauses. Normal people run on from time to time, (bracket in lots of asides), or otherwise vary their cadence and rhythm within clauses more than LLMs tend to.

Section headings that are intended to be "cute" and "snappy" or "impactful" rather than technically correct or compact: this is especially a tell when the cuteness/impactfulness is deeply mismatched with the seriousness or technical depth of the subject matter.

Horrible trite analogies that show no actual real understanding of the actual logical, mathematical, or visuo-spatial relationships involved. I.e. analogies are based on linguistic semantics, and not e.g. mathematical isomorphism or core dynamics. "Humans cannot fly. Building airplanes does not change that; it only means we built a machine that flies for us". Can't imagine a more retarded and useless analogy for something as complex as the article topic.

Verbose repetition: The article defines two workarounds: "tool use" and "agentic" orchestration, then defines them, then in the paragraph immediately following, says the exact same thing. There are basically multiple (small paragraphs) that all say nothing at all more than the sentence "LLMs do not reliably perform long, exact computations on their own, so in practice we often delegate the execution to external tools or orchestration systems".

Pseudo-profound bullshit: (https://doi.org/10.1017/S1930297500006999). E.g. "A system that cannot compute cannot truly internalize what computation is." There is thankfully not too much of this in the article, and it appears mostly early on.

Missing key / basic logic (or failing to mention such points clearly) when this would be strongly expected by any serious practitioner or expert: E.g. in this article, we should have seen some simple nice centered LaTeX showing the scaled dot-product self attention equation, and then some simple notation to represent the `.chunk` call, and subsequent linear projection, something like H = [H1 | H2], or etc., I shouldn't have to squint at two small lines of PyTorch code to find this. It should be clear immediately this model is not trained, and this is essentially just compiling a VM into a Transformer, and not revealed more clearly only at the end.

Re: Executing programs inside transformers with exponentially faster inference

#103

Post by itigges22 »

teiferer wrote: post #75

Why would that be desirable? If we take the human brain as an example, it's pretty bad at computation. Multiply two 10-digit numbers takes forever, despite the enormous size of its neural network. It's not the right tool for the job - a few deterministic logic gates could do that much more efficiently. That same circuit can't do much else, but multiplying, oh boy, it's good at that! Why do we think that artificial ne…

[flagged]

Re: Executing programs inside transformers with exponentially faster inference

#104

Post by itigges22 »

btown wrote: post #17

This seems way cooler than just computation (which is easy to hand off to a tool, and arguably more predictable that way). The broader point here is that you can have your model switch dynamically to/from a kind of attention that scales with the log of the token count, by only exploring the convex hull in a 2D space. A less capable version of attention, to be sure, but one capable of tracing a program’s execution wit…

[flagged]

Re: Executing programs inside transformers with exponentially faster inference

#105

Post by D-Machine »

dwa3592 wrote: post #67
YeGoblynQueenne wrote: post #62

Earlier quoted context omitted.

I can't see anything about "training a transformer". I'm trying to understand if e.g. the Sudoku solver was learned from examples (in which case, what examples?) or whether it was manually coded and then "compiled" into weights.

I assumed that they had to train, otherwise how else would they get "inside" a transformer. I also feel a bit of bad smell from the article. Sounding revolutionary with no details or clear explanation.

There is no training in the usual sense of the term, i.e. no gradient descent, no differentiable loss function. They use deceptive language early on to make it sound this way, but near the end make it clear their model as is isn't actually differentiable, and in theory might still work if made differentiable. But they don't actually know.

But IMO this is BS because I don't know how one would get or generate training data, or how one would define a continuous loss function that scores partially-correct / plausible outputs (e.g. is a "partially correct" program / algorithm / code even coherent, conceptually).

Re: Executing programs inside transformers with exponentially faster inference

#106

Post by gavinray »

YeGoblynQueenne wrote: post #59

So, what I'm trying to understand, and I can't find any clear information about that in the article, is how they "compiled" e.g. the Sudoku solver into a Transformer's weights. Did they do it manually? Say, they took the source of a hand-coded Sudoku solver and put it through their code-to-weight compiler, and thus compiled the code to the Transformer weights? Or did they go the Good, Old-Fashioned, Deep Learning way…

The article states they trained a WASM interpreter and programs are represented as WASM bytecode

Re: Executing programs inside transformers with exponentially faster inference

#107

Post by D-Machine »

manas96 wrote: post #43

LLMs are not deterministic per my understanding. A program always produces the same output for the same input and instructions (ignore FP accuracy for now). How is determinism achieved here?

LLMs (or at least transformer-based LLMs) are effectively almost entirely deterministic, the randomness being largely only present due to (unnecessary) optimizations and other tweaks.

Temperature is not at all core to LLMs, it is something that rather makes the outputs more varied and desirable for human consumption generally. It is trivial to set to zero for applications like this.

On CPUs, the models are essentially fully deterministic, even with FP accuracy, and most common kernels have reproducible (albeit slower) variants even on GPUs. Otherwise, yes, FP non-associativity on GPUs is the only real source of randomness in inference.

The other issue arises from batch invariance, but this is a problem that occurs only at scale when serving multiple users / inputs have some randomness too. You can (usually) trivially eliminate this by controlling what goes in the batch or making the batch size be one. There are also other more clever mitigations for this, none of which are secrets.

EDIT - Forgot reference: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Re: Executing programs inside transformers with exponentially faster inference

#108

Post by casey2 »

YAY! this is exactly what I wanted as the final step of some agent batching prompts to sub agents, but seeing it in action made me realize: Wow being able to talk to any program during runtime, including the OS, because an LLM is your CPU! What a concept!

Computing is going to be so weird in a few decades, writing programs faster than I can speak with full semantic introspection into every byte of code.

Re: Executing programs inside transformers with exponentially faster inference

#109

Post by D-Machine »

SPascareli13 wrote: post #84

If the model is trained to be a interpreter, then that means that the loss should reach 0 for it to be fully trained? Also, if it's execution is purely deterministic, you probably don't need non linearity in the layers, right?

The model isn't trained, it isn't differentiable (read carefully to the end: they say their model might still work if they made it differentiable, but they don't know), and it isn't clear IMO it could ever be made trainable (what is your loss function that scores a "partially correct" program / compiler, and how are you getting such training data?).

You need non-linearity in self-attention because it encodes feature / embedding similarities / correlations (e.g. self-attention is kernel smoothing) and/or multiplicative interactions, it has nothing to do with determinism/indeterminism. Also, LLMs are not really nondeterministic in any serious way, that all just comes from tweaks and optimizations that are not at all core to the architecture.

Re: Executing programs inside transformers with exponentially faster inference

#110

Post by D-Machine »

gavinray wrote: post #106
YeGoblynQueenne wrote: post #59

So, what I'm trying to understand, and I can't find any clear information about that in the article, is how they "compiled" e.g. the Sudoku solver into a Transformer's weights. Did they do it manually? Say, they took the source of a hand-coded Sudoku solver and put it through their code-to-weight compiler, and thus compiled the code to the Transformer weights? Or did they go the Good, Old-Fashioned, Deep Learning way…

The article states they trained a WASM interpreter and programs are represented as WASM bytecode

Nope, they encoded or compiled in a simple VM / WASM interpreter to the transformer weights, there is no training. You'd be forgiven for this misreading, as they deliberately mislead early on that their model is (in principle) trainable, but later admit that their actual model is not actually differentiable, but that a differentiable approximation "should" still work (despite no info about what loss function or training data could allow scoring partially correct / incomplete program outputs).
Post reply on HN