I trained a small transformer in 1.5hrs and it beats many LLMs • Hacker News Board

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#181

Post by qsera » Thu, Sep 03, 2026, 7:53 AM UTC

Dylan16807 wrote: post #180

qsera wrote: post #179

Earlier quoted context omitted.

> It reaches a new balance point with more doctors than before. But you just pull this "more doctors than before" out of your ass. There is no such assurance. If less people want to become doctors, then eventually there will be less doctors, not more.

I didn't pull it out of my ass. It's basic logic that if you remove a supply restriction then the supply goes up. Let's attach fake numbers: Right now only 50 people can get residency each year, and 80 people starting college each year want to be doctors and would work hard enough. If you let everyone get residency, at first you'd make 80 doctors per year. This would decrease doctor pay and then only 75 people would…

> But if this ever dropped back near 50 for long..

There is no reason why it would bounce back at 50. What if it bounce back at around 20 and settle around 30? Then you have less doctors than before...

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#182

Post by Dylan16807 » Thu, Sep 03, 2026, 6:17 PM UTC

qsera wrote: post #181

Dylan16807 wrote: post #180

Earlier quoted context omitted.

I didn't pull it out of my ass. It's basic logic that if you remove a supply restriction then the supply goes up. Let's attach fake numbers: Right now only 50 people can get residency each year, and 80 people starting college each year want to be doctors and would work hard enough. If you let everyone get residency, at first you'd make 80 doctors per year. This would decrease doctor pay and then only 75 people would…

> But if this ever dropped back near 50 for long.. There is no reason why it would bounce back at 50. What if it bounce back at around 20 and settle around 30? Then you have less doctors than before...

We already know that when the doctor production is 50, the number of potential doctors settles at 80.

Supply and demand means any lower doctor production causes even higher wages and that draws in even more potential doctors.

By what mechanism would it settle at 30? In particular, if it would settle at 30 despite an even harsher doctor shortage, why is it not already at or below 30?

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#183

Post by m4nu3l » Sat, Sep 05, 2026, 1:02 PM UTC

dfdydx wrote: post #50

pj_mukh wrote: post #44

Earlier quoted context omitted.

>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?

Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.

Also, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions). Transformers are just the ones that are most common and successful today.