Nice to see arc-agi-1 framed this way — I'd been circling the same idea without the right words.
I trained a small transformer in 1.5hrs and it beats many LLMs
41–50 of 182 posts
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#42> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training.
I saw this on the community note for the last blog you wrote - anything to do here.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#43Earlier quoted context omitted.
I studied biochem in undergrad and my classes were full of premed students. I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds. I've never looked at doctors the same way again after college. I looked up to them as a child,…
>When we should let doctors from overseas immigrate and easily become practicing doctors here in the US. Lol, what does this has to do with anything regarding aptitude or curiosity!?
To the parent posters credit, he's very honest that he developed a personal complex against doctors when he was a poor student. He just sees it as a way to lash out at American doctors, the irony being that foreign medical graduates leaving their families and communities to practice in America largely do so because they are exceptionally money motivated. That doesn't make them bad doctors, but there's certainly less likelihood they're doing it purely for the love of medicine or a desire to care for their communities.
I do agree with the sentiment though that the US needs to fund more residency slots as it's an asinine professional barrier, and that we would benefit from more physicians coming from more diverse financial backgrounds.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#44Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…
>> NOT an LLM. its a small ar transforme
Super cool project! Though, aren't most modern LLM's ar transformers internally?
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#45Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…
>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?
It's an slm
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#46Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…
Fyi, the link to rhabdomyolysis is broken on the homepage! The URL is repeated
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#47I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop?
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#48Earlier quoted context omitted.
>When we should let doctors from overseas immigrate and easily become practicing doctors here in the US. Lol, what does this has to do with anything regarding aptitude or curiosity!?
To the parent posters credit, he's very honest that he developed a personal complex against doctors when he was a poor student. He just sees it as a way to lash out at American doctors, the irony being that foreign medical graduates leaving their families and communities to practice in America largely do so because they are exceptionally money motivated. That doesn't make them bad doctors, but there's certainly less…
> certainly less likelihood they're doing it purely for the love of medicine or a desire to care for their communities.
Maybe self preservation?
In east europe, public hospitals will force doctors to work 36 hours shifts (overnight ER with theoretical sleep). Doctors have a full criminal liability for mall practise.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#49Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
Crazy, considering rhabdo isn't that rare.
This was in India, where he describes the medical knowledge of providers as subpar at best.
Re: I trained a small transformer in 1.5hrs and it beats many LLMs
#50Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but use…
>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.