Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smallest transformer that can add two 10-digit numbers

github.com

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Smallest transformer that can add two 10-digit numbers

#31

Post by ratmeadow »

Lerc wrote: post #28
Sophira wrote: post #19

I get that this is technically interesting, for certain, but the sheer amount of energy and associated global warming risk needed to do something with >=99% accuracy that we've been able to do easily for decades with a guaranteed 100% accuracy seems to me to be wasteful to the extreme.

What would be an acceptable amount of energy to spend on something that someone has done in a different manner before? Would you rather we stick with all of the current known ways to do things. Does this boil down to a condemnation of all scientific endeavours if they use resources? Would it change things if the people who did it enjoyed themselves? Would they have spent more energy playing a first person shooter to…

Ok I don't really care either way but to play devil's advocate, what exactly is this specific challenge of adding numbers with a transformer model demonstrating/advancing? The pushpack from people, albeit a little aggressive, does have a grain of truth. We're demonstrating that a model which uses preexisting addition instructions can add numbers? I mean yeah you can do it with arbitrarily few parameters because you don't need a machine learning model at all. Not exactly groundbreaking so I reckon the debate is fair.

Now if you said this proof of addition opens up some other interesting avenue of research, sure.

Re: Smallest transformer that can add two 10-digit numbers

#32

Post by ratmeadow »

delta_p_delta_x wrote: post #23

Very cool, but can I suggest the `add` CPU instruction instead? Supports 64-bit numbers, and it's encoded in hardware, and no need to cross a PCIe interface into a beefy, power-hungry GPU and back again. And chances are it's cross-platform, because basically every ISA since the very first has had `add`.

"smallest supercomputing cluster that can add two numbers"

Re: Smallest transformer that can add two 10-digit numbers

#33

Post by Lerc »

alexlitz wrote: post #26
sowbug wrote: post #25

Earlier quoted context omitted.

I ask this question as someone who can't do much more than confirm that your blog post is written in English by someone who knows math. Does this result suggest that if we had N clever humans manually building an LLM, they might come up with something as smart as a frontier model, but potentially 45 times smaller? (1644 / 36 ~= 45, N = very large, time not specified)

I imagine getting things to be polysemantic in a way that does not interfere would lead to sublinear scaling. Also there are smaller ones that were trained so would still be more like 311/36 ~= 8.6.

>I imagine getting things to be polysemantic in a way that does not interfere would lead to sublinear scaling.

True, but with even smarter humans, you could exploit the interactions for additional calculations.

While it sounds a bit silly, it is one of the hypotheses behind a fast takeoff. An AI that is sufficiently smart could design a network better than a trained one and could make something much smarter than itself on the same hardware. The question then becomes if that new smarter one can do an even better job. I suspect diminishing returns, but then again I am insufficiently smart.

Re: Smallest transformer that can add two 10-digit numbers

#34

Post by nurettin »

delta_p_delta_x wrote: post #23

Very cool, but can I suggest the `add` CPU instruction instead? Supports 64-bit numbers, and it's encoded in hardware, and no need to cross a PCIe interface into a beefy, power-hungry GPU and back again. And chances are it's cross-platform, because basically every ISA since the very first has had `add`.

I mean, yeah, no need to put a bunch of high powered cars in a circular track to watch them race really close to each other at incredible speeds, causing various hazards, either. Especially since city buses have been around for ages.

Re: Smallest transformer that can add two 10-digit numbers

#35

Post by Lerc »

ratmeadow wrote: post #31
Lerc wrote: post #28

Earlier quoted context omitted.

What would be an acceptable amount of energy to spend on something that someone has done in a different manner before? Would you rather we stick with all of the current known ways to do things. Does this boil down to a condemnation of all scientific endeavours if they use resources? Would it change things if the people who did it enjoyed themselves? Would they have spent more energy playing a first person shooter to…

Ok I don't really care either way but to play devil's advocate, what exactly is this specific challenge of adding numbers with a transformer model demonstrating/advancing? The pushpack from people, albeit a little aggressive, does have a grain of truth. We're demonstrating that a model which uses preexisting addition instructions can add numbers? I mean yeah you can do it with arbitrarily few parameters because you d…

>what exactly is this specific challenge of adding numbers with a transformer model demonstrating/advancing?

Well for starters, it puts the lie to the argument that a transformer can only output examples it has seen before. Performing the calculation on examples that haven't been seen demonstrates generalisation of the principles and not regurgitation.

While this misconception persists in a large number of people, counterexamples can always serve a useful purpose.

Re: Smallest transformer that can add two 10-digit numbers

#38

Post by qsera »

Lerc wrote: post #35
ratmeadow wrote: post #31

Earlier quoted context omitted.

Ok I don't really care either way but to play devil's advocate, what exactly is this specific challenge of adding numbers with a transformer model demonstrating/advancing? The pushpack from people, albeit a little aggressive, does have a grain of truth. We're demonstrating that a model which uses preexisting addition instructions can add numbers? I mean yeah you can do it with arbitrarily few parameters because you d…

>what exactly is this specific challenge of adding numbers with a transformer model demonstrating/advancing? Well for starters, it puts the lie to the argument that a transformer can only output examples it has seen before. Performing the calculation on examples that haven't been seen demonstrates generalisation of the principles and not regurgitation. While this misconception persists in a large number of people, co…

>it puts the lie to the argument

But it does not, right? You can either show it something, or modify the parameters in a way that resemble the result of showing it something.

You can claim that the model didn't see the thing, but that would mean nothing, because you are making the same effect with parameter tweaks indirectly.

Re: Smallest transformer that can add two 10-digit numbers

#39

Post by vicchenai »

The leaderboard framing is clever - forces apples-to-apples comparison on a task where you can verify correctness deterministically. What I find interesting is the architectural constraints: 10-digit addition requires maintaining ~20 digits of working state across the carry chain, which is fundamentally sequential. The fact that tiny transformers can learn this at all (rather than just memorizing) suggests they are finding some form of positional carry representation in their attention patterns. Would love to see ablations on how attention head count vs depth trade off here - my intuition is that carry propagation needs depth more than width.

Re: Smallest transformer that can add two 10-digit numbers

#40

Post by delta_p_delta_x »

nurettin wrote: post #34
delta_p_delta_x wrote: post #23

Very cool, but can I suggest the `add` CPU instruction instead? Supports 64-bit numbers, and it's encoded in hardware, and no need to cross a PCIe interface into a beefy, power-hungry GPU and back again. And chances are it's cross-platform, because basically every ISA since the very first has had `add`.

I mean, yeah, no need to put a bunch of high powered cars in a circular track to watch them race really close to each other at incredible speeds, causing various hazards, either. Especially since city buses have been around for ages.

I would similarly criticise a race car being used to do a city bus' job of getting a lot of people from point A to B.

Although the converse would be interesting, racing city buses.

Post reply on HN