Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

Post reply on HN
Print view View original

131–140 of 183 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#131

Post by westurner »

evilmathkid wrote: post #33
westurner wrote: post #26

How does it perform on ARC-AGI-3? There was this a few weeks ago: "Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163 >> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol What does that do with 5.6 Luna instead of the expensive models? What of 'schema' would improve the performanc…

its not gonna do well on ARC-3 without some significant changes and effort The new arch in that video is kinda misleading. Didn't really compare against proper baselines

Yeah, but still, how does that agent perform if paired with this inexpensively trained transformer instead of the more expensive foundation models?

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#132

Post by fc417fc802 »

kgwgk wrote: post #105
shsshs wrote: post #91

Earlier quoted context omitted.

[flagged]

> The distinction is meaningless IMO. Not when discussing whether it's an LLM. The second L in LLM does have a meaning.

I'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#133

Post by fc417fc802 »

Retric wrote: post #117
shsshs wrote: post #91

Earlier quoted context omitted.

[flagged]

Words and symbols alone don’t make a language. Zipper and and? and coin is pimple, Ocean! run bumper.

Your example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language?

When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability.

To come at it from a slightly different angle - does compiled binary code count as a language? If not, why not? (I'll suggest that it's a language albeit not a natural one.)

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#134

Post by andai »

jrflo wrote: post #55
bee_rider wrote: post #47

I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning b…

The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems. The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the datase…

Does ARC measure "one shot learning"? I heard that the major unsolved problem in ML was developing systems that are good at dealing with novel problems.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#135

Post by suddenlybananas »

10xDev wrote: post #94
rappatic wrote: post #86

I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.

No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.

Why not both?

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#136

Post by Dylan16807 »

embedding-shape wrote: post #9
f311a wrote: post #5

Earlier quoted context omitted.

The whole point of his model is to optimize for a very specific benchmark. BUT, he does not use labels when training, so the model does not know the answers.

> The whole point of his model is to optimize for a very specific benchmark. But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?

Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#137

Post by evilmathkid »

ozim wrote: post #115
evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

*Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho* On specific tasks sure on "generic performance" whatever that means for you not really.

As of today yes I agree with you

in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#138

Post by evilmathkid »

tasuki wrote: post #106
evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

As an aside, the anagram is so good!

haha thanks!

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#139

Post by Retric »

fc417fc802 wrote: post #133
Retric wrote: post #117

Earlier quoted context omitted.

Words and symbols alone don’t make a language. Zipper and and? and coin is pimple, Ocean! run bumper.

Your example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language? When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability. To come at it from a slightly differe…

Any sequence could be meaningful or meaningless depending on the grammar involved. Open a word document in photoshop and the program has no idea what it’s dealing with.

The reason the language term in LLM is meaningful is how the training, symbol mapping, etc is designed around human languages. The model doesn’t process raw text, instead there’s a critical processing step which allows the magic to happen.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#140

Post by foota »

evilmathkid wrote: post #129
foota wrote: post #98

> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is possible Guarantees no synthetic data can be used It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like…

train from scratch only during the 12 hours allowed on Kaggle Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun

[dead]
Post reply on HN