Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

31–40 of 182 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#31
post #4

> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training. That's why I like when a new set of tests like a new AR…

> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.

I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt)

And more compute increases the perf (after dealing with other scaling problems)

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#32

Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that? If you just wanted to pass these specific tasks in this specific ben…

[dead]

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#33

How does it perform on ARC-AGI-3? There was this a few weeks ago: "Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163 >> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol What does that do with 5.6 Luna instead of the expensive models? What of 'schema' would improve the performanc…

its not gonna do well on ARC-3 without some significant changes and effort

The new arch in that video is kinda misleading. Didn't really compare against proper baselines

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#34

Earlier quoted context omitted.

Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

Out of interest, would you call BERT an LLM? It’s pre trained but not particularly large.

idk the definition is fuzzy. thats why people use the "modern" qualifier to talk about decoder-only style and this is also not clean since you now have reasoning models which are separate

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#36
post #14
post #11

Earlier quoted context omitted.

Crazy, considering rhabdo isn't that rare.

“What do you call a medical student who graduated at the bottom of their class?” “Doctor.”

I studied biochem in undergrad and my classes were full of premed students.

I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds.

I've never looked at doctors the same way again after college. I looked up to them as a child, yet after seeing how the sausages were made, I started to doubt everything.

I frequently ask doctors, who spend all of ten minutes with me while the nurses do all the work, about the molecular specifics of what they're talking about. They talk down to me as if they're explaining to a child, yet they're frequently quite wrong. I'm not trying to sound superior to them, but I'm shocked they seem to care so little about the subject. It doesn't give me much hope about what they know and their abilities or competency.

I suspect surgeons and specialists are a different breed and aren't like this at all.

And to be clear, this isn't everyone. But it does seem to be the majority I've interacted with throughout my life.

When they act disgruntled at patient interaction, I detest that their profession tries to cap the number of med students per year. We should be letting in as many med students as we can take. We should let doctors from overseas immigrate and easily become practicing doctors here in the US. We should provide easy paths for nurses to become doctors.

The premed students in my university were chiefly concerned about money and prestige. They drove BMWs gifted to them by their parents and laughed at what I drove and how hard I studied. I had to put up with their bullying for years. I know not everyone who studies to become a doctor is like that, but it permanently skewed my view of their profession.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#37
post #16

Earlier quoted context omitted.

Nowhere does he say he built an llm. Hes using a transformer, not an llm.

> Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LL…

Transformer solves a Seq2Seq problem just like RNNs. All Seq2Seq problems need not involve a language. In this case teaching on ARC puzzles doesn't mean what he trained is now trained on a language which will be English(or any other language) in this case. So, does his training successfully models "English as a language" -> No. This implies it is not "Large" and has not modeled any "language".

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#38
post #36
post #14

Earlier quoted context omitted.

“What do you call a medical student who graduated at the bottom of their class?” “Doctor.”

I studied biochem in undergrad and my classes were full of premed students. I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds. I've never looked at doctors the same way again after college. I looked up to them as a child,…

>When we should let doctors from overseas immigrate and easily become practicing doctors here in the US.

Lol, what does this has to do with anything regarding aptitude or curiosity!?

Post reply on HN