Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

Post reply on HN
Print view View original

161–170 of 183 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#161

Post by docheinestages »

nullbio wrote: post #92
docheinestages wrote: post #52

Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.

Why would that be cheating? That's what humans do when they learn, they look for the signals and patterns that reduce the possible set of answers so they can converge on the solution and narrow the search space.

I mean, I'm interested to know if the frontier models also get to see all questions at once. Then it's more fair game than if they just see one question at a time.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#162

Post by usernametaken29 »

I think you’re asking the right questions, sample inefficiency is horrible in modern LLMs. Despite this, I saw your analysis:

> The biggest increases in scores were due to

Modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm, etc.) More data diversity, better shuffling of data scaling up: 8 layers instead of 4

This is commonly called squeezing the lemon and is usually a bit of a last resort. You should be able to achieve near SoTa with your new method, before you squeeze any lemons. This is, because the old SoTa is typically not using new optimisers and thus your results will be distorted by a large margin.

In terms of sample efficiency I want to add two things:

Runtime per-puzzle fine tuning is a very good target that provides a LOT of information. People have not looked at evolutionary methods to harness induction since the 90ies - if I was to work on ARC ever again I’m fairly certain this is where I’d look.

Best of luck, padawan

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#163

Post by fc417fc802 »

Retric wrote: post #158
fc417fc802 wrote: post #149

Earlier quoted context omitted.

Is the same processing step not happening here? To my mind the argument against this model qualifying as a language model is that while the sequence of tokens may technically qualify as a sort of language it doesn't appear to be generalized by any reasonable interpretation. Further, the model doesn't appear to be able to handle unstructured inputs and outputs in the "language" - everything seems to be highly structur…

The distinction is meaningful because the process described here isn’t bound by the same constraints, resulting in meaningful consequences. Suppose we flipped the initial stings and fed that into the process. There would still be meaning to extract from the training set but our new Reversed English but it is not English so the preprocessing step can’t be based on that assumption.

I don't understand what point you're trying to make here. At the risk of being tangential (due to not understanding) LLMs have famously been able to accept base64 encoded natural language as input "out of the box" because (it seems) under the hood they already transparently translate all inputs into their own internal "language" system on the fly. Presumably a similar translation could be learned for input of reverse english (or rather merely adjusting attention) although at a glance it seems like output might not be feasible.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#164

Post by TZubiri »

evilmathkid wrote: post #146
evilmathkid wrote: post #109

Earlier quoted context omitted.

> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam " What you described is cheating because more time than the exam permits. ARC was designed specifically…

I didn't phrase that well and cant edit, so clarifying: What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam "

>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

>What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"

This is getting into metaphorgotten territory, I'd have to do a deep dive to really understand whether it is "cheating" or, more formally, a methodology that encourages overfitting.

But for what is worth, taking the textbook to the exam (regardless of whether you were born there or not), would be cheating in a test as well. Although it is possible that in the model training context it does not lead to overfitting, it certainly doesn't preclude it.

>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

It is comforting that someone agrees, to this day this is considered a gray-area (or not even that) by students, graduates and those that look up to the University of Buenos Aires institution.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#165

Post by nomel »

xdavidliu wrote: post #89
rappatic wrote: post #86

I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.

the caveat is that we are not learning those small number of samples from scratch, since we're coming in with a large amount of training already, much of it from before we were even born

I've seen a few videos about people having their girlfriend/dad/etc play very simple video games for the very first time. It's a really great example of this. The most basic, simple, mechanics are difficult for them, and take surprising repetition to remember, because they have zero framework around any of it. It's really similar to watching a genetic algorithm learn to play Mario, but on fast forward.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#166

Post by nomel »

matteoraso wrote: post #145
10xDev wrote: post #94

Earlier quoted context omitted.

No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.

>We never stop learning, we don't have a "training phase". We kind of do, if you count critical periods in childhood.

Probable mechanism: "What we found is a rapid increase in GABA in children, associated with learning, ..." [1]

[1] https://www.brown.edu/news/2022-11-15/children-learning

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#167

Post by [deleted] »

evilmathkid wrote: post #18

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…

[deleted]

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#168

Post by FridgeSeal »

_ache_ wrote: post #103
bkaae wrote: post #61

Earlier quoted context omitted.

There is no language in the training of this, so there is no l.

In computer science, that is technically a language. A formal language if you want to look it up on Wikipedia.

Technically correct, but not in a functionally useful way.

The “L” in LLM’s generally refers to human-language specifically. You’d expect to feed it…human text. Nitpicking that the human text also constitutes a mathematical language is like, correct, but so general as to be unhelpful.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#169

Post by evilmathkid »

TZubiri wrote: post #164
evilmathkid wrote: post #146

Earlier quoted context omitted.

I didn't phrase that well and cant edit, so clarifying: What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam "

>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam" >What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam" This is getting into metaphorgotten territory, I'd have to do a deep dive to…

well not if its an open textbook exam

in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc

this is one such case

--

the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#170

Post by porridgeraisin »

dpoloncsak wrote: post #84
pj_mukh wrote: post #44

Earlier quoted context omitted.

>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?

Pretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations. Transformers are what really started the LLM Boom, and seem to be crucial to the technology. They also have other applications, such as what OP created

In my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper.

Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed.

Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.

Post reply on HN