Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trained a small transformer in 1.5hrs and it beats many LLMs

mvakde.github.io

Post reply on HN
Print view View original

171–180 of 183 posts

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#171

Post by JBits »

docheinestages wrote: post #161
nullbio wrote: post #92

Earlier quoted context omitted.

Why would that be cheating? That's what humans do when they learn, they look for the signals and patterns that reduce the possible set of answers so they can converge on the solution and narrow the search space.

I mean, I'm interested to know if the frontier models also get to see all questions at once. Then it's more fair game than if they just see one question at a time.

They look at each problem individually.

As the ARC AGI guidelines [1] state, 'a core design principle of ARC-AGI is that the test taker must not know what the test will be.'

[1] https://arcprize.org/policy#dataset-security

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#172

Post by qsera »

Dylan16807 wrote: post #130
qsera wrote: post #38

Earlier quoted context omitted.

>When we should let doctors from overseas immigrate and easily become practicing doctors here in the US. Lol, what does this has to do with anything regarding aptitude or curiosity!?

If we reduce the artificial shortage of doctors then the number of people entering the field specifically to make big money goes down.

> people entering the field specifically to make big money goes down...

This is a self defeating argument, because for it to work it requires that there is a larger number of people who love the field than the number of people who are in only for the $$$ (or else there will be even more scarcity of doctors)..

Which is not the case.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#175

Post by dfdydx »

fc417fc802 wrote: post #132
kgwgk wrote: post #105

Earlier quoted context omitted.

> The distinction is meaningless IMO. Not when discussing whether it's an LLM. The second L in LLM does have a meaning.

I'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).

Are number sequences a language?

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#176

Post by fc417fc802 »

dfdydx wrote: post #175
fc417fc802 wrote: post #132

Earlier quoted context omitted.

I'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).

Are number sequences a language?

Are rectangles squares? I can express a passage written in english as a sequence of numbers (ie tokens).

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#177

Post by Dylan16807 »

qsera wrote: post #172
Dylan16807 wrote: post #130

Earlier quoted context omitted.

If we reduce the artificial shortage of doctors then the number of people entering the field specifically to make big money goes down.

> people entering the field specifically to make big money goes down... This is a self defeating argument, because for it to work it requires that there is a larger number of people who love the field than the number of people who are in only for the $$$ (or else there will be even more scarcity of doctors).. Which is not the case.

We have a surplus of people that want to become doctors and would work hard enough to do so. If we removed the artificial limits on residency, the supply of doctors would increase, which lowers wages somewhat, which lowers the number of people that want to become doctors. It reaches a new balance point with more doctors than before.

That itself is a good thing.

For it to also improve the motivations of doctors, all that needs to be true is that someone more motivated by money is more likely to be in the group that switches career.

It does not require the number of people that love the field to be a majority. Not that loving the field is a binary in the first place. People are a mix of motivations and cutoffs are arbitrary.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#178

Post by TZubiri »

evilmathkid wrote: post #169
TZubiri wrote: post #164

Earlier quoted context omitted.

>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam" >What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam" This is getting into metaphorgotten territory, I'd have to do a deep dive to…

well not if its an open textbook exam in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc this is one such case -- the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute

I think you can do whatever you want with your train set, what would be concerning is training with the test set.

I don't think overfitting is separate, the consequence of putting the benchmarks in your training set is not that you "cheat" by breaking some moral code, it's that it breaks the purpose of the benchmark and trains your model to be good at that benchmark only, instead of being generally useful.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#179

Post by qsera »

Dylan16807 wrote: post #177
qsera wrote: post #172

Earlier quoted context omitted.

> people entering the field specifically to make big money goes down... This is a self defeating argument, because for it to work it requires that there is a larger number of people who love the field than the number of people who are in only for the $$$ (or else there will be even more scarcity of doctors).. Which is not the case.

We have a surplus of people that want to become doctors and would work hard enough to do so. If we removed the artificial limits on residency, the supply of doctors would increase, which lowers wages somewhat, which lowers the number of people that want to become doctors. It reaches a new balance point with more doctors than before. That itself is a good thing. For it to also improve the motivations of doctors, all t…

> It reaches a new balance point with more doctors than before.

But you just pull this "more doctors than before" out of your ass. There is no such assurance. If less people want to become doctors, then eventually there will be less doctors, not more.

Re: I trained a small transformer in 1.5hrs and it beats many LLMs

#180

Post by Dylan16807 »

qsera wrote: post #179
Dylan16807 wrote: post #177

Earlier quoted context omitted.

We have a surplus of people that want to become doctors and would work hard enough to do so. If we removed the artificial limits on residency, the supply of doctors would increase, which lowers wages somewhat, which lowers the number of people that want to become doctors. It reaches a new balance point with more doctors than before. That itself is a good thing. For it to also improve the motivations of doctors, all t…

> It reaches a new balance point with more doctors than before. But you just pull this "more doctors than before" out of your ass. There is no such assurance. If less people want to become doctors, then eventually there will be less doctors, not more.

I didn't pull it out of my ass. It's basic logic that if you remove a supply restriction then the supply goes up.

Let's attach fake numbers: Right now only 50 people can get residency each year, and 80 people starting college each year want to be doctors and would work hard enough. If you let everyone get residency, at first you'd make 80 doctors per year. This would decrease doctor pay and then only 75 people would want to be doctors, then 70. But if this ever dropped back near 50 for long, wages would spike back up, and the candidates would hit 80 again. It stabilizes in between, at 63.

With a bottleneck at X, and potential supply at Y, removing the bottleneck gets you a number between X and Y. All else equal, it can't stabilize below X. There will not 'eventually' be less doctors.

Alternatively don't even remove the bottleneck at first. Increase the residency bottleneck to 60. The number of potential doctors drops below 80, but it stays above 60, so now you have 60 doctors per year and the most money-motivated took a different major.

TL;DR: Let the number of residencies rise and the number of people that want to be doctors lower until they meet in the middle. Because residencies decide the actual number of doctors, meeting in the middle gives you more doctors.

Post reply on HN