Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
Crazy, considering rhabdo isn't that rare.
I trained a small transformer in 1.5hrs and it beats many LLMs
11–20 of 182 posts
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#12"I don’t understand why others didn’t figure this out"
- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#13Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#14Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
Crazy, considering rhabdo isn't that rare.
“What do you call a medical student who graduated at the bottom of their class?”
“Doctor.”
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#15Earlier quoted context omitted.
The whole point of his model is to optimize for a very specific benchmark. BUT, he does not use labels when training, so the model does not know the answers.
> The whole point of his model is to optimize for a very specific benchmark. But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations.
Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#16Earlier quoted context omitted.
I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days. I found it to be a very interesting angle.
> He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people…
Nowhere does he say he built an llm. Hes using a transformer, not an llm.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#17> Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised too
I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#18Hi! Author here. Surprised to see this on HN now. Happy to answer any questions!
Some context about this:
- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs
- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.
- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)
- Can the perf be improved? Yes but the competition is ongoing so can't talk about it
- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho
- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#19Earlier quoted context omitted.
> He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people…
Nowhere does he say he built an llm. Hes using a transformer, not an llm.
> Nowhere does he say he built an llm. Hes using a transformer, not an llm.
Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results".
Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?