Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

301–310 of 354 posts

Re: Microgpt

#301

Post by radarsat1 »

chongli wrote: post #296
geokon wrote: post #141

> What’s the deal with “hallucinations”? The model generates tokens by sampling from a probability distribution. It has no concept of truth, it only knows what sequences are statistically plausible given the training data. Extremely naiive question.. but could LLM output be tagged with some kind of confidence score? Like if I'm asking an LLM some question does it have an internal metric for how confident it is in its…

Having a confidence score isn't as useful as it seems unless you (the user) know a lot about the contents of the training set. Think of traditional statistics. Suppose I said "80% of those sampled preferred apples to oranges, and my 95% confidence interval is within +/- 2% of that" but then I didn't tell you anything about how I collected the sample. Maybe I was talking to people at an apple pie festival? Who knows!…

I think your last point raises the following question: how would you change your answer if you know they read all about guns and death and how one causes the other? What if they'd seen pictures of guns? And pictures of victims of guns annotated as such? What if they'd seen videos of people being shot by guns?

I mean I sort of understand what you're trying to say but in fact a great deal of knowledge we get about the world we live in, we get second hand.

There are plenty of people who've never held a gun, or had a gun aimed at them, and.. granted, you could argue they probably wouldn't read that line the same way as people who have, but that doesn't mean that the average Joe who's never been around a gun can't enjoy media that features guns.

Same thing about lots of things. For instance it's not hard for me to think of animals I've never seen with my own eyes. A koala for instance. But I've seen pictures. I assume they exist. I can tell you something about their diet. Does that mean I'm no better than an LLM when it comes to koala knowledge? Probably!

Re: Microgpt

#303

Post by subset »

OJFord wrote: post #184
subset wrote: post #121

Earlier quoted context omitted.

Andrej Karpathy has a walkthrough blog post here: https://karpathy.github.io/2026/02/12/microgpt/

That is the article being discussed.

Gosh, tired brain moment apologies. I thought it'd linked to the original code gist.

Re: Microgpt

#304

Post by matheusmoreira »

Jolter wrote: post #273
matheusmoreira wrote: post #266

Earlier quoted context omitted.

> One of us is an advanced autocomplete engine. The other is a human, capable of making judgements on what is conscious and what is not. What evidence is there that your "judgements" are anything other than advanced autocompletion? Concepts introduced into a self-training wetware CPU via its senses over a lifetime in order to predict tokens and form new concepts via logical manipulation? > Your philosophizing about s…

Surely you’re not seriously saying that you believe AI agents, in their current state of the art, meet whatever criteria you have for being ”alive”? That’s kind of how you’re coming across. I don’t really know how to respond to that, because it’s so preposterous.

I'm saying you, a human, are not as special as you think you are.

Re: Microgpt

#305

Post by grey-area »

etothet wrote: post #227

Even if you have some basic understanding of how LLMs work, I highly recommend Karpathy’s intro to LLMs videos on YouTube. - https://m.youtube.com/watch?v=7xTGNNLPyMI - https://m.youtube.com/watch?v=EWvNQjAaOHw

Thanks, this is very very long but very good background on how production LLMs work.

Re: Microgpt

#306

Post by chongli »

radarsat1 wrote: post #301
chongli wrote: post #296

Earlier quoted context omitted.

Having a confidence score isn't as useful as it seems unless you (the user) know a lot about the contents of the training set. Think of traditional statistics. Suppose I said "80% of those sampled preferred apples to oranges, and my 95% confidence interval is within +/- 2% of that" but then I didn't tell you anything about how I collected the sample. Maybe I was talking to people at an apple pie festival? Who knows!…

I think your last point raises the following question: how would you change your answer if you know they read all about guns and death and how one causes the other? What if they'd seen pictures of guns? And pictures of victims of guns annotated as such? What if they'd seen videos of people being shot by guns? I mean I sort of understand what you're trying to say but in fact a great deal of knowledge we get about the…

It’s more complicated to think about, but it’s still the same result. Think about the structure of a dictionary: all of the words are defined in terms of other words in the dictionary, but if you’ve never experienced reality as an embodied person then none of those words mean anything to you. They’re as meaningless as some randomly generated graph with a million vertices and a randomly chosen set of edges according to some edge distribution that matches what we might see in an English dictionary.

Bringing pictures into the mix still doesn’t add anything, because the pictures aren’t any more connected to real world experiences. Flooding a bunch of images into the mind of someone who was blind from birth (even if you connect the images to words) isn’t going to make any sense to them, so we shouldn’t expect the LLM to do any better.

Think about the experience of a growing baby, toddler, and child. This person is not having a bunch of training data blasted at them. They’re gradually learning about the world in an interactive, multi-sensory and multi-manipulative manner. The true understanding of words and concepts comes from integrating all of their senses with their own manipulations as well as feedback from their parents.

Children also are not blank slates, as is popularly claimed, but come equipped with built-in brain structures for vision, including facial recognition, voice recognition (the ability to recognize mom’s voice within a day or two of birth), universal grammar, and a program for learning motor coordination through sensory feedback.

Re: Microgpt

#308

Post by smj-edison »

Somewhat unrelated, but the generated names are surprisingly good! They're certainly more sane then appending -eigh to make a unique name.

Re: Microgpt

#309

Post by love2read »

teleforce wrote: post #94

Someone has modified microgpt to build a tiny GPT that generates Korean first names, and created a web page that visualizes the entire process [1]. Users can interactively explore the microgpt pipeline end to end, from tokenization until inference. [1] English GPT lab: https://ko-microgpt.vercel.app/

By "modified" this person of course means that they swapped out the list of X0,000 names from English to Korean names. That is seemingly the only change.

The attached website is a fully ai-generated "visualization" based on the original blog post with little added.

Post reply on HN