Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing programs inside transformers with exponentially faster inference

percepta.ai

Post reply on HN
Print view View original

111–120 of 139 posts

Re: Executing programs inside transformers with exponentially faster inference

#111

Post by D-Machine »

vuciuc wrote: post #73
mobilejdral wrote: post #31

Earlier quoted context omitted.

I spent the entire time reading it pondering the same thing. 1. The article presents that calling out to a tool like python is "expensive" because of the overhead of forking a process, loading up the python env etc, but why not just eliminate that overhead and embed WebAssembly so this "tool call" is near zero? This feels very similar to the discussion in the 90's around the overhead of threads v.s. processes or kern…

> "A system that cannot compute cannot truly internalize what computation is." The way this is formulated, almost sounds like they think that giving llms this ability will bring them closer to having experiences of computation or smth? Weird?

One of the worst sentences in the article, clear example of pseudo-profound bullshit, almost certainly LLM-generated.

Re: Executing programs inside transformers with exponentially faster inference

#112

Post by D-Machine »

bee_rider wrote: post #60

Is their convex hull attention mechanism new and generally useable? I mean, it substantially restricts the shape of the model, so it isn’t a universal solution of course, but it does seem to overcome a pretty annoying limitation.

If you read the section "Richer attention mechanisms", you can see, no, the mechanism is not generally useable (it requires significant modification to become differentiable). They later speculate:

    While we do not yet know whether exact softmax attention
    can be maintained with the same efficiency, it is easy to
    approximate it with k-sparse softmax attention: retrieve
    the top-k keys and perform the softmax only over those
but if you have played around with training models that use e.g. topk or other hard thresholding operations in e.g. PyTorch (or just think about how many gradients become zero with such an operation) you know that these tend to work only in extremely limited / specific cases, and make training even more finicky than it already is.

Re: Executing programs inside transformers with exponentially faster inference

#113

Post by que3216 »

For me this felt key to understanding the value vs tool calls:

> Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.

Re: Executing programs inside transformers with exponentially faster inference

#114

Post by bee_rider »

D-Machine wrote: post #112
bee_rider wrote: post #60

Is their convex hull attention mechanism new and generally useable? I mean, it substantially restricts the shape of the model, so it isn’t a universal solution of course, but it does seem to overcome a pretty annoying limitation.

If you read the section "Richer attention mechanisms", you can see, no, the mechanism is not generally useable (it requires significant modification to become differentiable). They later speculate: While we do not yet know whether exact softmax attention can be maintained with the same efficiency, it is easy to approximate it with k-sparse softmax attention: retrieve the top-k keys and perform the softmax only over t…

I saw that, but the image included nearby made it look like it might be plausible to replace the 1D line around their points with a pretty narrow 2D area. This could still be a somewhat effective filter, right?

Re: Executing programs inside transformers with exponentially faster inference

#115

Post by xpl »

Is this a joke? If it's not trainable / differentiable when why do it in the first place? It's just as inefficient and inflexible as it gets compared to tool calling — you have to statically bake programs in the weights, model cannot introspect it and modify, it has very limited IO capabilities, bad performance, bad everything. Its like a weird brainfuck-esque VM — cool that you can do it, but for what except some lulz?

But maybe it's just too genius and I don't understand it.

Re: Executing programs inside transformers with exponentially faster inference

#116

Post by D-Machine »

bee_rider wrote: post #114
D-Machine wrote: post #112

Earlier quoted context omitted.

If you read the section "Richer attention mechanisms", you can see, no, the mechanism is not generally useable (it requires significant modification to become differentiable). They later speculate: While we do not yet know whether exact softmax attention can be maintained with the same efficiency, it is easy to approximate it with k-sparse softmax attention: retrieve the top-k keys and perform the softmax only over t…

I saw that, but the image included nearby made it look like it might be plausible to replace the 1D line around their points with a pretty narrow 2D area. This could still be a somewhat effective filter, right?

The problem is they are talking about tricks for compiling VMs into transformer weights, which is basically unrelated to actually training transformers on data via gradient descent. Once you get into this actual messy practical reality, you have non-trivial stuff like sparsemax and the Gumbel-Softmax trick to get some desirable improvements to things like the softmax, without all the gradient destruction of things like top-k approaches, but usually at pretty serious other costs (most approaches using Gumbel-Softmax I have read essentially create a bi-level optimization problem that is claimed to be "solved" by some handwavey annealing, but which is clearly highly unstable and hard to tune. I don't know if things have improved here since I last read on it).

So the issue isn't if there aren't ways to effectively approximate their approach, from a strictly numerical approximation standpoint, it is that other factors matter much more in optimization when training on actual data.

Re: Executing programs inside transformers with exponentially faster inference

#117

Post by D-Machine »

que3216 wrote: post #113

For me this felt key to understanding the value vs tool calls: > Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.

Except their process isn't actually differentiable, as they admit near the end of the post, they just sort hand-wavily suggest that approximately differentiable methods "should" work. Also no mention at all of what the training data would be, where it would come from, or how a loss function could be constructed to continuously score "partially correct" programs (of what that would even mean, or if that idea is even coherent).

What was a good point, mentioned by @hedgehog in this thread (https://news.ycombinator.com/item?id=47367986), is that tool-calls break batching a lot, so there could be huge efficiency gains at scale if you can just pass through a computation sub-network (even if that sub-net is frozen and can't be updated, and is programmed in manually rather than trained in).

Why on Earth you'd want that sub-net to be a clunky transformer rather than just an efficient, GPU-accelerated custom non-trainable layer, though, is unclear to me.

Re: Executing programs inside transformers with exponentially faster inference

#118

Post by D-Machine »

xpl wrote: post #115

Is this a joke? If it's not trainable / differentiable when why do it in the first place? It's just as inefficient and inflexible as it gets compared to tool calling — you have to statically bake programs in the weights, model cannot introspect it and modify, it has very limited IO capabilities, bad performance, bad everything. Its like a weird brainfuck-esque VM — cool that you can do it, but for what except some lu…

I'd tend to agree, the only good points I've seen were made by @hedgehog [1] here in this thread:

    I'm not sure about the rest but a significant problem with high frequency tool calling (especially in training) is that it breaks batching.
and then later by @ACCount37 [2]:

    I'm less interested in turning programs into transformers and more interested in turning programs into subnetworks within large language models.
In theory, if you can create a very efficient sub-net to replicate certain tool calls (even if the weights are frozen during any training steps, and manually compiled), this might help with making inference much more efficient at scale. No idea why in general you would want to do this through the clunky transformer architecture though. Just implement a non-trainable, GPU-accelerated layer to do the compute and avoid the tool-call.

[1] https://news.ycombinator.com/item?id=47367986

[2] https://news.ycombinator.com/item?id=47363909

Re: Executing programs inside transformers with exponentially faster inference

#119

Post by alexlitz »

dnautics wrote: post #69

the paper is burying the lede here (i think?) > The key technical unlock is to restrict lookup heads to head dimension 2, which enables a decoding path where the dominant retrieval/update operations can be computed in log time in the sequence length (for this structured executor regime), rather than by a full prefix-sized attention sweep. edit: i understand how hullkv works now. very clever. I dont understand why thi…

The buried lede is this, if you have two dimensions and use rope, and hard-max attention you could simply store addresses as a given theta. With RoPE and sufficient precision that pretty easily gets you relative addressing with just one head and absolute with three (treating BOS as a sink getting rotation relative to it with orthogonal unit queries then using the result to counter rotate your own relative position with the complex conjugate). With less precision just add a few more heads with different thetas.

Re: Executing programs inside transformers with exponentially faster inference

#120

Post by dnautics »

D-Machine wrote: post #98
dnautics wrote: post #90

Earlier quoted context omitted.

> lookup tables anymore, that was never a good analogy in the first place good analogy otherwise, wasn't hash tables the motivation for the kv tables?

Well, one can never be sure what the real motivation for a lot of DL advances, as most papers are post-hoc obscurantism / hand-waving or even just outright nonsense (see: internal covariate shift explanations for batch norm, which arguably couldn't be more wrong https://arxiv.org/pdf/1805.11604 ). When you really get into this stuff, you tend to see the real motivations as either e.g. kernel smoothing (see comments /…

yeah now that I think about this, I think the hull-kv will not scale to comprehension beyond ~simple computational tasks.
Post reply on HN