Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

181–190 of 354 posts

Re: Microgpt

#181

Post by amelius »

Lionga wrote: post #146
geokon wrote: post #141

> What’s the deal with “hallucinations”? The model generates tokens by sampling from a probability distribution. It has no concept of truth, it only knows what sequences are statistically plausible given the training data. Extremely naiive question.. but could LLM output be tagged with some kind of confidence score? Like if I'm asking an LLM some question does it have an internal metric for how confident it is in its…

The LLM has an internal "confidence score" but that has NOTHING to do with how correct the answer is, only with how often the same words came together in training data. E.g. getting two r's in strawberry could very well have a very high "confidence score" while a random but rare correct fact might have a very well a very low one. In short: LLM have no concept, or even desire to produce of truth

> In short: LLM have no concept, or even desire to produce of truth

They do produce true statements most of the time, though.

Re: Microgpt

#182

Post by allovertheworld »

znnajdla wrote: post #95

Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…

It just doesn’t work that way, LLMs need to be generalised a lot to be useful even in specific tasks.

It really is the antithesis to the human brain, where it rewards specific knowledge

Re: Microgpt

#183

Post by jaen »

amelius wrote: post #181
Lionga wrote: post #146

Earlier quoted context omitted.

The LLM has an internal "confidence score" but that has NOTHING to do with how correct the answer is, only with how often the same words came together in training data. E.g. getting two r's in strawberry could very well have a very high "confidence score" while a random but rare correct fact might have a very well a very low one. In short: LLM have no concept, or even desire to produce of truth

> In short: LLM have no concept, or even desire to produce of truth They do produce true statements most of the time, though.

That's just because true statements are more likely to occur in their training corpus.

Re: Microgpt

#184

Post by OJFord »

subset wrote: post #121
red_hare wrote: post #56

This is beautiful and highly readable but, still, I yearn for a detailed line-by-line explainer like the backbone.js source: https://backbonejs.org/docs/backbone.html

Andrej Karpathy has a walkthrough blog post here: https://karpathy.github.io/2026/02/12/microgpt/

That is the article being discussed.

Re: Microgpt

#185

Post by borplk »

Can anyone mention how you can "save the state" so it doesn't have to train from scratch on every run?

Re: Microgpt

#186

Post by OJFord »

retube wrote: post #137

Can you train this on say Wikipedia and have it generate semi-sensible responses?

If you increase all the numbers (including, as a result, the time to train).

Re: Microgpt

#187

Post by kilroy123 »

jimbokun wrote: post #47

It’s pretty staggering that a core algorithm simple enough to be expressed in 200 lines of Python can apparently be scaled up to achieve AGI. Yes with some extra tricks and tweaks. But the core ideas are all here.

I strongly suspect we're like 4 more elegant algorithms away from a real AGI.

Re: Microgpt

#188

Post by regularfry »

kelnos wrote: post #108
charcircuit wrote: post #80

Earlier quoted context omitted.

It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.

That's not learning, though. That's just taking new information and stacking it on top of the trained model. And that new information consumes space in the context window. So sure, it can "learn" a limited number of things, but once you wipe context, that new information is gone. You can keep loading that "memory" back in, but before too long you'll have too little context left to do anything useful. That kind of cap…

Two things:

1. It's still memory, of a sort, which is learning, of a sort. 2. It's a very short hop from "I have a stack of documents" to "I have some LoRA weights." You can already see that happening.

Re: Microgpt

#189

Post by ricericerice »

mootothemax wrote: post #180
otabdeveloper4 wrote: post #176

Earlier quoted context omitted.

> BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. And we had good language models before BERT too. They were a royal bitch to train properly, though. Nowadays you can get the same with just 30 minutes of prompt engineering.

> > BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. Astute readers will note what’s been missed here. Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.

After some research, I think I understand what you're getting at here - BERT being a model for encoding text but not architecturally feasible to generate text with it, which "LLMs" (the lack of definition here is resulting in you two talking past eachother), maybe more accurately referred to as GPTs, can do.

Also the irony of your comment when it in itself was confidently stated yet void of any content was not missed either - consider dropping the superiority complex next time.

Re: Microgpt

#190

Post by regularfry »

Dansvidania wrote: post #106
charcircuit wrote: post #80

Earlier quoted context omitted.

It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.

That’s not learning. That’s carrying over context that you are trusting is correctly summarised over from one conversation to the next.

Which sounds uncomfortably like human memory, which gets rewritten from one recollection to the next. Somehow, we cope.
Post reply on HN