Just wanted to say that the rhabdo part mentioned on your site was really impressive! Speaking as a medical doctor and full stack engineer myself.
I trained a small transformer in 1.5hrs and it beats many LLMs
51–60 of 182 posts
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#52Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#53Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…
[dead]
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#54I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning b…
Basically, you have a bunch of Q,A pairs in the training dataset. Here, it was trained to next-word predict the question itself, as well as next-word predict the answer given the question as prompt. This is bog-standard, no one's complaining.
In the test dataset's Q,A pairs, it was only trained to next-word predict the question itself, and it was not given the answer at all.
It was then evaluated by seeing if it is able to output A_test given the Q_test as prompt.
What would be cheating is training it to produce A_test (given Q_test as prompt) as well, since then you can always make a model that scores 100% by just memorising Q_test, A_test pairs.
The complaints online mostly stem from not reading that properly and assuming they trained on Q_test,A_test instead of just Q_test. This is further because these days large LLMs are inadvertently trained on many benchmark solutions even unintentionally due to the massive scale of data and the infeasibility of auditing it all. But none of that is the case here.
The reason you want to train on Q_test is because in these AR transformer models, they learn useful composable encodings of Q by simply learning to next-word predict Q. So you enable the model to learn composable encodings of the test questions, so that it can hopefully "connect it" to an earlier train problem it had seen, and adapt the solution it had seen for that, much like humans do in school exams.
Without this step, you are making it difficult for the model to "connect" the test question to a train question it had seen earlier, and then it still has to adapt the solution. This way, you precompute that "this test question is like this train question" and then during the exam you only have to do the adapting the solution part after a simpler "retrieval" process.
You can just think of next-word training Q_test as a "retrieval" process.
This practice often used in continual learning or "test time training" is not yet useful in general real world ML tasks due to the differences in memory and compute requirements, and more so the general fragility of training large neural networks in a streaming realtime way (as opposed to large data, batched), versus inferencing from a static neural network. It is due to that fragility that I believe (correct me if I am wrong) this guy had to train on a batch of Q_tests. If you enforced that you will not provide Q_test_2 before they answer Q_test_1, the performance will drop.
While the increased compute and memory is difficult to solve inherently, there are various efforts being made to fix the fragility, especially in reinforcement learning where this is called "streaming RL", there is revival of interest as seen in RLC 2026.
[Note] Arc-AGI-1 doesn't have any actual english words or such, but it's simpler to pretend it was a basic Q&A benchmark to explain the above
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#55I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning b…
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems.
The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the dataset that's used to update the weights according to your loss function, you are "encoding" the patterns from the training set directly into your model. The eval set is what you use to track performance while training, it is NOT used to update model weights, but shows how well the model generalizes. The test set is a private holdout set that is only used when you're "done" developing your model. The difference between test and eval is information leakage: you can use performance against the eval set to modify your hyperparameters and model architecture to get better eval scores. So while the eval set doesn't directly update the weights, it can indirectly cause "overfitting" by tailoring your model to do well on the eval set. What you really want to see is the private test set performance, not the eval set. For all we know, this model could be ridiculously overfit on the eval set and perform poorly on the private test set.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#56Earlier quoted context omitted.
Crazy, considering rhabdo isn't that rare.
Everyone and their dog who is on statins knows what rhabdo is. Bonkers!
I've been on statins for years, and I don't remember anyone talking to me about rhabdo. To be fair the education I received about my medications was a firehose of information after a heart attack and major heart surgery, so perhaps it's possible I missed a few things.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#57Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…
>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?
[dead]
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#58I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning b…
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems. The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the datase…
They have _not_ trained on the test set.
On the private test set, the right way to evaluate this type of model, is giving i it the test question Q, which it will first train to AR predict first, and then it will inference using the just-updated weights with Q as prompt, giving you back A, and then you compare A with A_true secretly.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#59[deleted]
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#60Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
I think is more concerning doctors didnt know about rhabdomyolysis...