Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

Post reply on HN
Print view View original

151–160 of 183 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#151

Post by bunderbunder »

alibero wrote: post #110
evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

Thank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task. Had a couple of questions: 1) You note that ARC-AGI is a meta-learning task, have you tried any meta-lear…

I spent some time working with that approach of using LLMs to generate synthetic labeled data for use in training more specialized models. It mostly didn't work.

The problem was that getting the LLM to generate training data that sufficiently resembled real-world data was labor intensive and expensive. More labor intensive and expensive, it turns out, than just using real data.

What worked better was using the LLM to label the training data. But even there we had to be careful about introducing weird biases.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#152

Post by evilmathkid »

piqufoh wrote: post #144
evilmathkid wrote: post #142

Earlier quoted context omitted.

2 different things are being called leakage here 1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training 2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser…

> weight update during eval: this is a form of test time training and not really cheating. Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.

no, most models on kaggle are finetuning during test time, (including LLM based approaches)

Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#153

Post by evilmathkid »

RoadieRoller wrote: post #150

I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him? https://news.ycombinator.com/item?id=49412396

Yeah but I posted this work back in Dec-Feb lol

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#154

Post by calebkaiser »

dpoloncsak wrote: post #84
pj_mukh wrote: post #44

Earlier quoted context omitted.

>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?

Pretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations. Transformers are what really started the LLM Boom, and seem to be crucial to the technology. They also have other applications, such as what OP created

It gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this:

A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B

Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#155

Post by RoadieRoller »

evilmathkid wrote: post #153
RoadieRoller wrote: post #150

I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him? https://news.ycombinator.com/item?id=49412396

Yeah but I posted this work back in Dec-Feb lol

My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way.

What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#156

Post by anon291 »

foota wrote: post #119
nicman23 wrote: post #116

Earlier quoted context omitted.

how would you as a human know the answer to the arc-agi questions?

I spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.

Predicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions.

> I don't think a baby or even average kid could solve these

The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#157

Post by evilmathkid »

RoadieRoller wrote: post #155
evilmathkid wrote: post #153

Earlier quoted context omitted.

Yeah but I posted this work back in Dec-Feb lol

My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way. What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.

Yeah agree with pg completely. Multiple people I know did this

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#158

Post by Retric »

fc417fc802 wrote: post #149
Retric wrote: post #139

Earlier quoted context omitted.

Any sequence could be meaningful or meaningless depending on the grammar involved. Open a word document in photoshop and the program has no idea what it’s dealing with. The reason the language term in LLM is meaningful is how the training, symbol mapping, etc is designed around human languages. The model doesn’t process raw text, instead there’s a critical processing step which allows the magic to happen.

Is the same processing step not happening here? To my mind the argument against this model qualifying as a language model is that while the sequence of tokens may technically qualify as a sort of language it doesn't appear to be generalized by any reasonable interpretation. Further, the model doesn't appear to be able to handle unstructured inputs and outputs in the "language" - everything seems to be highly structur…

The distinction is meaningful because the process described here isn’t bound by the same constraints, resulting in meaningful consequences.

Suppose we flipped the initial stings and fed that into the process. There would still be meaning to extract from the training set but our new Reversed English but it is not English so the preprocessing step can’t be based on that assumption.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#160

Post by 5555watch »

It means the questions including their answers are dependent. Ie, theres a data generating process for them, that the model uncovers. Like a KNN is known to have near Bayes accuracy as k/n to 0, n to infty, k to infty. The data reveals the dgp.

I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.

Post reply on HN