Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

Post reply on HN
Print view View original

21–30 of 183 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#21

Post by kuczmama »

evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#22

Post by evilmathkid »

embedding-shape wrote: post #19
dakolli wrote: post #16

Earlier quoted context omitted.

Nowhere does he say he built an llm. Hes using a transformer, not an llm.

> Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LL…

Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs.

When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks

And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#23

Post by soVeryTired »

embedding-shape wrote: post #19
dakolli wrote: post #16

Earlier quoted context omitted.

Nowhere does he say he built an llm. Hes using a transformer, not an llm.

> Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LL…

It’s neither large nor language-based. ARC-AGI-1 is grid-based and nonverbal.

Use of a transformer is not necessary or sufficient to qualify as an LLM.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#24

Post by soVeryTired »

evilmathkid wrote: post #22
embedding-shape wrote: post #19

Earlier quoted context omitted.

> Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LL…

Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

Out of interest, would you call BERT an LLM? It’s pre trained but not particularly large.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#25

Post by imdsm »

qlm wrote: post #11
xeonax wrote: post #3

Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)

Crazy, considering rhabdo isn't that rare.

Everyone and their dog who is on statins knows what rhabdo is. Bonkers!

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#26

Post by westurner »

How does it perform on ARC-AGI-3?

There was this a few weeks ago:

"Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163

>> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol

What does that do with 5.6 Luna instead of the expensive models?

What of 'schema' would improve the performance of mdlARC?

mdlARC: https://github.com/mvakde/mdlARC

There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#27

Post by stymaar »

soVeryTired wrote: post #24
evilmathkid wrote: post #22

Earlier quoted context omitted.

Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

Out of interest, would you call BERT an LLM? It’s pre trained but not particularly large.

AFAIK, the “large” qualifier came when transformers allowed to scale the size of language models compared to the recurrent models that where in fashion before. And although BERT isn't large by today's standard, it was large enough for the time.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#28

Post by yorwba »

embedding-shape wrote: post #19
dakolli wrote: post #16

Earlier quoted context omitted.

Nowhere does he say he built an llm. Hes using a transformer, not an llm.

> Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LL…

A LLM should at the very least be a language model, i.e. be able to take human-readable text as input or produce it as output. Transformers are used for plenty of tasks that don't involve language, for example object detection or blind source separation, where the models aren't called LMs; and on the other hand there are some LLM architectures that exclusively use linear attention variants and aren't really transformers anymore.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#29

Post by evilmathkid »

kuczmama wrote: post #21
evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?

Yeah I've reached much higher perf but

- it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me

- more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#30

Post by asabla »

evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

Thank you for answering these questions. Looking forward for the next write up about this.
Post reply on HN