201–210 of 354 posts
I tried building a tiny model last weekend, but it was very difficult to find any articles that weren’t broken ai slop.
Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…
You've just reinvented machine learning
Is there a similarly simple implementation with tensorflow? I tried building a tiny model last weekend, but it was very difficult to find any articles that weren’t broken ai slop.
Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…
Karpathy has other projects, e.g. : https://github.com/karpathy/nanochat
You can train a model with GPT-2 level of capability for $20-$100.
But, guess what, that's exactly what thousands of AI researchers have been doing for the past 5+ years. They've been training smallish models. And while these smallish models might be good for classification and whatnot, people strongly prefer big-ass frontier models for code generation.
Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…
It just doesn’t work that way, LLMs need to be generalised a lot to be useful even in specific tasks. It really is the antithesis to the human brain, where it rewards specific knowledge
Since this post is about art, I'll embed here my favorite LLM art: the IOCCC 2024 prize winner in bot talk, from Adrian Cable ( https://www.ioccc.org/2024/cable1/index.html ), minus the stdlib headers: #define a(_)typedef _##t #define _(_)_##printf #define x f(i, #define N f(k, #define u _Pragma("omp parallel for")f(h, #define f(u,n)for(I u=0;u c[t*W+k]*y[i*W+k];v[h]= _*X->f[t]*l[i]+!!i*v[h];}x D-c)i[r]+=v[i];}I main…
wiat what does this do?
It’s pretty staggering that a core algorithm simple enough to be expressed in 200 lines of Python can apparently be scaled up to achieve AGI. Yes with some extra tricks and tweaks. But the core ideas are all here.
LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.
LLMs are artificial general intelligence, as per the Wikipedia definition:
> generalise knowledge, transfer skills between domains, and solve novel problems without task‑specific reprogramming
Even GPT-3 could meet that bar.
Earlier quoted context omitted.
A 16 year old has been training for almost 16 years to drive a car. I would argue the opposite: Waymo’s / Specific AIs need far less data than humans. Humans can generalize their training, but they definitely need a LOT of training!
No 16 year old has practiced driving a car for 16 years.
Earlier quoted context omitted.
> AGI just means human level intelligence. It seems more like people haven't decided on what the goal post is. If AGI is just another human, that's pretty underwhelming. That's why people are imagining something that surpasses humans by heaps and bounds in terms of reasoning, leading to wondrous new discoveries.
"Just another human" would be outright astonishing if it landed.