Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

101–110 of 354 posts

Re: Microgpt

#101

Post by npn »

znnajdla wrote: post #95

Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…

what gut? we are already doing that. there are a lot of "tiny" LLMs that are useful: M$ Phi-4, Gemma 3/3n, Qwen 7B... There are even smaller models like Gemma 270M that is fine tuned for function calls.

they are not flourish yet because of the simple reason: the frontier models are still improving. currently it is better to use frontier models than training/fine-tuning one by our own because by the time we complete the model the world is already moving forward.

heck even distillation is a waste of time and money because newer frontier models yield better outputs.

you can expect that the landscape will change drastically in the next few years when the proprietary frontier models stop having huge improvements every version upgrade.

Re: Microgpt

#102

Post by otabdeveloper4 »

znnajdla wrote: post #95

Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…

We had good small language models for decades. (E.g. BERT)

The entire point of LLMs is that you don't have to spend money training them for each specific case. You can train something like Qwen once and then use it to solve whatever classification/summarization/translation problem in minutes instead of weeks.

Re: Microgpt

#103

Post by with »

"everything else is just efficiency" is a nice line but the efficiency is the hard part. the core of a search engine is also trivial, rank documents by relevance. google's moat was making it work at scale. same applies here.

Re: Microgpt

#104

Post by otabdeveloper4 »

charcircuit wrote: post #80
0xbadcafebee wrote: post #74

Earlier quoted context omitted.

It still can't learn. It would need to create content, experiment with it, make observations, then re-train its model on that observation, and repeat that indefinitely at full speed. That won't work on a timescale useful to a human. Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. So we're hyper-tweaking LLMs to make them more useful faster, in th…

It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.

> they update their memories

Their contexts, not their memories. An LLM context is like 100k tokens. That's a fruit fly, not AGI.

Re: Microgpt

#105

Post by znnajdla »

npn wrote: post #101
znnajdla wrote: post #95

Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…

what gut? we are already doing that. there are a lot of "tiny" LLMs that are useful: M$ Phi-4, Gemma 3/3n, Qwen 7B... There are even smaller models like Gemma 270M that is fine tuned for function calls. they are not flourish yet because of the simple reason: the frontier models are still improving. currently it is better to use frontier models than training/fine-tuning one by our own because by the time we complete t…

I’ve tried those tiny LLMs and they don’t seem useful to me for real world tasks. They are toys for super simple autocomplete.

Re: Microgpt

#106

Post by Dansvidania »

charcircuit wrote: post #80
0xbadcafebee wrote: post #74

Earlier quoted context omitted.

It still can't learn. It would need to create content, experiment with it, make observations, then re-train its model on that observation, and repeat that indefinitely at full speed. That won't work on a timescale useful to a human. Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. So we're hyper-tweaking LLMs to make them more useful faster, in th…

It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.

That’s not learning. That’s carrying over context that you are trusting is correctly summarised over from one conversation to the next.

Re: Microgpt

#107

Post by lukan »

with wrote: post #103

"everything else is just efficiency" is a nice line but the efficiency is the hard part. the core of a search engine is also trivial, rank documents by relevance. google's moat was making it work at scale. same applies here.

Sure, but understanding the core concepts are essential to make things efficient and as far as I understand, this has mainly educational purposes ( it does not even run on a GPU).

Re: Microgpt

#108

Post by kelnos »

charcircuit wrote: post #80
0xbadcafebee wrote: post #74

Earlier quoted context omitted.

It still can't learn. It would need to create content, experiment with it, make observations, then re-train its model on that observation, and repeat that indefinitely at full speed. That won't work on a timescale useful to a human. Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. So we're hyper-tweaking LLMs to make them more useful faster, in th…

It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.

That's not learning, though. That's just taking new information and stacking it on top of the trained model. And that new information consumes space in the context window. So sure, it can "learn" a limited number of things, but once you wipe context, that new information is gone. You can keep loading that "memory" back in, but before too long you'll have too little context left to do anything useful.

That kind of capability is not going to lead to AGI, not even close.

Re: Microgpt

#109

Post by lukan »

hackersk wrote: post #83

[flagged]

"The math makes so much more sense when you implement it yourself vs reading papers."

Something I found to be universal true when dealing with math. My brain pretty much refuses to learn abstract math concepts in theory, but applying them with a practical problem is a very different experience for me (I wish school math would have had a bigger focus on practical applications).

Re: Microgpt

#110

Post by Dansvidania »

jimbokun wrote: post #85
cellis wrote: post #71

Earlier quoted context omitted.

A 16 year old has been training for almost 16 years to drive a car. I would argue the opposite: Waymo’s / Specific AIs need far less data than humans. Humans can generalize their training, but they definitely need a LOT of training!

No 16 year old has practiced driving a car for 16 years.

If you see gaining fine motor control, understanding pictographic language […] as a prerequisite to driving a car, then yes, all of them are
Post reply on HN