Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

Post reply on HN
Print view View original

101–110 of 183 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#101

Post by TZubiri »

>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.

I disagree, but i agree that training with answers is worse.

In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing.

It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well.

It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases.

Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do.

And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other.

Hell, I'm not surprised that the software that the mass of test -studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#102

Post by evilmathkid »

bbor wrote: post #62
evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

First: this is really great technical writing, especially when you get into the rebuttals. Firm & clear without polemics -- props, and thanks for open-sourcing! That said; I don't have the time, energy, or anywhere near the expertise to challenge you on the DL specifics, but I feel compelled to add another voice to the chorus of doubters nonetheless. Using other ARC examples at runtime (effectively, yes?) for "transd…

Thanks!

I think this is a great question. I have some thoughts on this but no hard evidence (neither does anyone else!)

Your argument relies on the AGI system being the model arch + weights. I think that the weights are irrelevant. The training algorithm is what is AGI: You choose/find a training set that covers a task, and then some form of deep learning with a big neural net.

For LLMs (which is a weak kind of AGI), this is freezing a model after NTP pretraining + RL postraining. This allows us to do well on a wide distribution of tasks.

But if we had a scoped task, I think its possible to just do the same thing with (1) a smaller dataset that covers this task + (2) not freezing the model (ie. test time training). This is easy for ARC because the data is small and has been curated well yes, but i don't see why can't this apply to more complex/ill-defined tasks (obv lots of things unsolved to make it work today)

> even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason.

In this hypothetical world, the dataset becomes incredibly large, and training on it makes it close to an LLM. You can then finetune transductively and we get the same thing (other approaches already do this iirc)

--

Note: BTW I'm not claiming that this is an AGI system, the benchmark creator also was clear that ARC is not sufficient for AGI (his goal was just to point out unsolved stuff, and the stuff ARC-2 pointed out was demonstrated very clearly by large reasoning models).

In the above work, I just wanted to show that AR transformers without pretraining can perform really well on this benchmark, which was incredibly non-obvious before.

The claims in this argument are separate and I haven't proved them yet

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#103

Post by _ache_ »

bkaae wrote: post #61
dnautics wrote: post #45

Earlier quoted context omitted.

It's an slm

There is no language in the training of this, so there is no l.

In computer science, that is technically a language. A formal language if you want to look it up on Wikipedia.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#104

Post by evilmathkid »

foota wrote: post #97

> I agree that its rare to see to face problem sets in real life where every problem is given at once. Even if it is (like an exam), humans can usually only attempt one at a time Just one small snippet that I thought was interesting. I would always read through ~the entire exam before starting. Both so that I could find the problems most approachable to me, but also because sometimes it helps me figure out the rest o…

same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#105

Post by kgwgk »

shsshs wrote: post #91
dfdydx wrote: post #50

Earlier quoted context omitted.

Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.

[flagged]

> The distinction is meaningless IMO.

Not when discussing whether it's an LLM. The second L in LLM does have a meaning.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#106

Post by tasuki »

evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

As an aside, the anagram is so good!

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#109

Post by evilmathkid »

TZubiri wrote: post #101

>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. I disagree, but i agree that training with answers is worse. In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was…

> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question

Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs).

(There's also access to experts who know the answer, which kaggle bans by banning the internet)

Lucas describes it well here (and his original tweet up the thread): https://x.com/giffmana/status/2002128356901597509

--

Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#110

Post by alibero »

evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

Thank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task.

Had a couple of questions:

1) You note that ARC-AGI is a meta-learning task, have you tried any meta-learning algorithms such as MAML?

2) Do you think this approach could extend to ARC-AGI 3? Or do you think the interactive environments require a higher level of complexity than what can be achieved with a small model?

Post reply on HN