Smallest transformer that can add two 10-digit numbers • Hacker News Board

Re: Smallest transformer that can add two 10-digit numbers

#91

Post by alexlitz » Sat, Feb 28, 2026, 5:19 PM UTC

Lerc wrote: post #33

alexlitz wrote: post #26

Earlier quoted context omitted.

I imagine getting things to be polysemantic in a way that does not interfere would lead to sublinear scaling. Also there are smaller ones that were trained so would still be more like 311/36 ~= 8.6.

>I imagine getting things to be polysemantic in a way that does not interfere would lead to sublinear scaling. True, but with even smarter humans, you could exploit the interactions for additional calculations. While it sounds a bit silly, it is one of the hypotheses behind a fast takeoff. An AI that is sufficiently smart could design a network better than a trained one and could make something much smarter than itse…

Yeah that is plausible enough.

Re: Smallest transformer that can add two 10-digit numbers

#92

Post by MarkusQ » Sat, Feb 28, 2026, 5:35 PM UTC

getnormality wrote: post #89

xg15 wrote: post #81

> Self-attention is required. The model must contain at least one self-attention layer. This is the defining feature of a transformer — without it, you have an MLP or RNN, not a transformer. I think it would be interesting to see challenges where two networks are trained and evaluated on the exact same datasets and the architecture is the same except for the presence of self-attention layers in one network. So far it…

> So far it seems to me that self-attention really brought new capabilities to a network Do we have a layman explanation for what makes self-attention so uniquely powerful? Something more than "it lets you do self-attention".

Computational power. Without self attention, you have a sloppy implementation of something called a PDA (push-down-automaton) -- like an old HP calculator. With it, you have an even sloppier implementation of a Turing machine.

So (modulo a _lot_ of details) it increases the power from that of a "calculator" to that of a "computer".

Re: Smallest transformer that can add two 10-digit numbers

#93

Post by undersuit » Sat, Feb 28, 2026, 5:46 PM UTC

famouswaffles wrote: post #54

prng2021 wrote: post #46

How is anyone predicting timelines for AGI when these systems can’t do basic addition of 2 arbitrary numbers with 100% accuracy?

Can you do basic addition of 2 arbitrary numbers with 100% accuracy (no tools) ? No you can't. You will make mistakes for a sufficiently large N even with pen and paper, and a very small N without. Are you no longer generally intelligent ?

Somewhere along the line a $10000 GPU has to be equivalent to using a finger to do arithmetic in the dust.

Re: Smallest transformer that can add two 10-digit numbers

#94

Post by subscribed » Sat, Feb 28, 2026, 5:52 PM UTC

tgv wrote: post #82

In the 90s, there were papers on emulating logical circuits with neurons. They would be bigger than this network, but at least always correct.

You might find https://corticallabs.com/cl1.html interesting (that's of course assuming this is not a scam, which I'm unable to assess).

Re: Smallest transformer that can add two 10-digit numbers

#95

Post by anthk » Sat, Feb 28, 2026, 9:15 PM UTC

Now, under T3X and Lisp under 64k:

https://t3x.org/lisp64k/index.html

Re: Smallest transformer that can add two 10-digit numbers

#96

Post by delta_p_delta_x » Sat, Feb 28, 2026, 9:49 PM UTC

ACCount37 wrote: post #50

delta_p_delta_x wrote: post #23

Very cool, but can I suggest the `add` CPU instruction instead? Supports 64-bit numbers, and it's encoded in hardware, and no need to cross a PCIe interface into a beefy, power-hungry GPU and back again. And chances are it's cross-platform, because basically every ISA since the very first has had `add`.

No. You cannot. It's the wrong tool for the problem. That little "add" of yours has the overhead of: having an LLM emit it as a tool call, having to pause the LLM inference while waiting for it to resolve, then having to encode the result as a token to feed it back. At the same time, a "transformer-native" addition circuit? Can be executed within a single forward pass at a trivial cost, generate transformer-native re…

I can't tell if this is satire or not. A1 top-tier.

Re: Smallest transformer that can add two 10-digit numbers

#97

Post by silentvoice » Sat, Feb 28, 2026, 10:24 PM UTC

I have no submission for this but I joined the hype in my own way by optimizing the training loop. These tiny models are not really well suited to frameworks like pytorch, and with highly patient AI agents we can now just inline the whole thing into C++ just to see what happens, which I do below:

https://www.reidatcheson.com/transformer/llm/ml/cuda%20graph...

Re: Smallest transformer that can add two 10-digit numbers

#98

Post by sureglymop » Sat, Feb 28, 2026, 10:34 PM UTC

medi8r wrote: post #66

Filligree wrote: post #30

Earlier quoted context omitted.

So the question is, why do we tokenise it in such a way that it makes everything harder?

There is no encoding that makes everything easier. You trade off maths for general intelligence. Now we are at a point where the LLM can just choose to use a normal calculator anyway!

Possibly unrelated but something I never fully understood: while we can't create a perfect parser for natural language, why don't we optimistically parse it to extract semantics and feed that into LLMs as well?

Re: Smallest transformer that can add two 10-digit numbers

#99

Post by nerdsniper » Sat, Feb 28, 2026, 11:57 PM UTC

varispeed wrote: post #90

SlinkyOnStairs wrote: post #79

Earlier quoted context omitted.

One major limitation of the LLM architecture is that even the failure mode varies unpredictably between inputs. The set of 11-digit numbers with any given failure mode (or even successful output) has no discernable pattern, merely whatever randomness the training process baked into the model. You can't predict ahead of time when they will fail spectacularly, nor draw a clear boundary around the failure cases. And ear…

Most humans struggle to compute 10 digit stuff. They use tools instead. Can LLM learn to use calculator? Sorry if that is a stupid question. Maybe brains are not well suited for calculations natively.

Yes. LLMs use calculators to great effect. More often, Python as a calculator.

Also, there exist autistic savants who prove that a human brain can be used to perform rote calculations on large numbers much faster than a human with a calculator can.

Re: Smallest transformer that can add two 10-digit numbers

#100

Post by wizzwizz4 » Sun, Mar 01, 2026, 11:39 AM UTC

18al wrote: post #59

kstrauser wrote: post #56

So, what happens when you test it on 11 digit numbers? I don’t mean that as a gotcha or “LOL dumb transformer” snark. More like, does the accuracy start to drop as you add digits? Or instead, maybe it’s the transformer equivalent of a stack overflow and it outputs a picture of a burning spoon or something? And for that matter, what’s it do with 9 digit numbers? Like, is it more accurate with them, or are these little…

Depends on how the transformer has been trained. If it has seen 11 digit examples while training it might work, else the input will be out of distribution and it will respond with a nonsensical number. For instance the current high score model (311 params [0]), when given 12345678900 + 1, responds with 96913456789. An interesting experiment would be: what's the minimum number of parameters required to handle unbounde…

If the neural network had moveable tape heads which could seek between invocations, and the inputs were provided in little-endian format, a fairly small model could implement arbitrary addition with carry, and you'd only need to add a few redundant dimensions to get something that could be trained.