Earlier quoted context omitted.
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
[flagged]
LLMs also don't use words
I trained a small transformer in 1.5hrs and it beats many LLMs
141–150 of 183 posts
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#142I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning b…
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems. The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the datase…
2 different things are being called leakage here
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#143Earlier quoted context omitted.
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems. The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the datase…
I am very new to this but applying human intuition this still feels like cheating. Knowing all the question that will be on the exam and working on understanding them even if you are never given answers will obviously give you and edge.
what you describe would be cheating. My approach is the opposite. What I did was "You are born during the exam, given access to a training set and the questions then learn from scratch during the exam"
I put more details in the answer here: https://news.ycombinator.com/item?id=49525841
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#144Earlier quoted context omitted.
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems. The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the datase…
2 different things are being called leakage here 1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training 2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser…
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#145I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.
No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.
>We never stop learning, we don't have a "training phase".
We kind of do, if you count critical periods in childhood.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#146>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. I disagree, but i agree that training with answers is worse. In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was…
> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam " What you described is cheating because more time than the exam permits. ARC was designed specifically…
I didn't phrase that well and cant edit, so clarifying:
What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#147Earlier quoted context omitted.
Sure, and those aren't LLMs?
[flagged]
I don't understand what you're saying? An LLM is a transformer model trained on a large corpus of natural language, often with some post-training. An image model is a different type of transformer model. What's controversial here?
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#148> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training. I saw this on the community note for the last blog you wrote - anything to do here.
Not true. This is allowed in a metalearning context. Its called transductive learning and has existed since the 90s: https://en.wikipedia.org/wiki/Transduction_(machine_learning...
I address this in more detail in the blog
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#149Earlier quoted context omitted.
Your example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language? When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability. To come at it from a slightly differe…
Any sequence could be meaningful or meaningless depending on the grammar involved. Open a word document in photoshop and the program has no idea what it’s dealing with. The reason the language term in LLM is meaningful is how the training, symbol mapping, etc is designed around human languages. The model doesn’t process raw text, instead there’s a critical processing step which allows the magic to happen.
Is the same processing step not happening here?
To my mind the argument against this model qualifying as a language model is that while the sequence of tokens may technically qualify as a sort of language it doesn't appear to be generalized by any reasonable interpretation. Further, the model doesn't appear to be able to handle unstructured inputs and outputs in the "language" - everything seems to be highly structured.
My line of reasoning could be approximately summarized as compiled binaries constituting a "real" (though not natural) language versus a sequential listing of chess positions that represent sequential game actions only being language "shaped".
Still, it's interesting to consider that if scaled up I expect the "repeat yourself" experiment would likely apply to the internal representation of the model in the same manner.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#150I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him?