Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt explained interactively

growingswe.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Microgpt explained interactively

#41

Post by [deleted] »

gwern wrote: post #40
growingswe wrote: post #29

Earlier quoted context omitted.

I had to look up what a content mill is. I'm not one, I think. It's "random" stuff because my interests are different. These posts are not written sequentially, I've been working on them (except for this MicroGPT one) for weeks and only publishing now.

Dude, you literally start the article with > Andrej Karpathy wrote a 200-line Python script that trains and runs a GPT from scratch, with no libraries or dependencies, just pure Python . Almost immediately afterwards, you have a section titled "Numbers, not letters". Need I go on? Interestingly, despite all the AI tics, the opening passes Pangram as 100% human... though all the following sections I randomly checked a…

[deleted]

Re: Microgpt explained interactively

#42

Post by mike_hearn »

windowshopping wrote: post #7

The part that eludes me is how you get from this to the capability to debug arbitrary coding problems. How does statistical inference become reasoning? For a long time, it seemed the answer was it doesn't. But now, using Claude code daily, it seems it does.

DNNs aren't really "statistical" inference in the way most people would understand the term statistics. The underlying maths owes much more to calculus than statistics. The model isn't just encoding statistics about the text it was trained on, it's attempting to optimize a solution to the problem of picking the next token with all the complexity that goes into that.

Re: Microgpt explained interactively

#43

Post by mike_hearn »

refulgentis wrote: post #22
love2read wrote: post #18

Is it becoming a thing to misspell and add grammatical mistakes on purpose to show that an LLM didn't write the blog post? I noticed several spelling mistakes in Karpathy's blog post that this article is based on and in this article.

People aren't gonna be happy I spell this out, but, Karpathy's not The Dude. He's got a big Twitter following so people assume somethings going on or important, but he just isn't. Biggest thing he did in his career was feed Elon's Full Self Driving delusion for years and years and years. Note, then, how long he lasted at OpenAI, and how much time he spends on code golf. If you're angry to read this, please, take a mi…

Tesla FSD isn't a delusion. There are people using it to successfully do long distance drives across the USA right now, without interventions. Dunno how much credit Karpathy gets for that, but the tech works.

Re: Microgpt explained interactively

#44

Post by jsheard »

jsheard wrote: post #9
ayhanfuat wrote: post #4

Earlier quoted context omitted.

You are absolutely right. The whole post reads like AI generated.

The rate they are posting new articles on random subjects is also a pretty indicative of a content mill. In 3 days they've covered machine learning, geometry, cryptography, file formats and directory services.

Addendum - now they've changed the dates of several articles retroactively, to increase the spacing.

Re: Microgpt explained interactively

#45

Post by 5o1ecist »

jsheard wrote: post #9
ayhanfuat wrote: post #4

Earlier quoted context omitted.

You are absolutely right. The whole post reads like AI generated.

The rate they are posting new articles on random subjects is also a pretty indicative of a content mill. In 3 days they've covered machine learning, geometry, cryptography, file formats and directory services.

[flagged]

Re: Microgpt explained interactively

#46

Post by love2read »

refulgentis wrote: post #22
love2read wrote: post #18

Is it becoming a thing to misspell and add grammatical mistakes on purpose to show that an LLM didn't write the blog post? I noticed several spelling mistakes in Karpathy's blog post that this article is based on and in this article.

People aren't gonna be happy I spell this out, but, Karpathy's not The Dude. He's got a big Twitter following so people assume somethings going on or important, but he just isn't. Biggest thing he did in his career was feed Elon's Full Self Driving delusion for years and years and years. Note, then, how long he lasted at OpenAI, and how much time he spends on code golf. If you're angry to read this, please, take a mi…

Is this AI generated?

Re: Microgpt explained interactively

#47

Post by refulgentis »

love2read wrote: post #46
refulgentis wrote: post #22

Earlier quoted context omitted.

People aren't gonna be happy I spell this out, but, Karpathy's not The Dude. He's got a big Twitter following so people assume somethings going on or important, but he just isn't. Biggest thing he did in his career was feed Elon's Full Self Driving delusion for years and years and years. Note, then, how long he lasted at OpenAI, and how much time he spends on code golf. If you're angry to read this, please, take a mi…

Is this AI generated?

What?

Re: Microgpt explained interactively

#48

Post by refulgentis »

mike_hearn wrote: post #43
refulgentis wrote: post #22

Earlier quoted context omitted.

People aren't gonna be happy I spell this out, but, Karpathy's not The Dude. He's got a big Twitter following so people assume somethings going on or important, but he just isn't. Biggest thing he did in his career was feed Elon's Full Self Driving delusion for years and years and years. Note, then, how long he lasted at OpenAI, and how much time he spends on code golf. If you're angry to read this, please, take a mi…

Tesla FSD isn't a delusion. There are people using it to successfully do long distance drives across the USA right now, without interventions. Dunno how much credit Karpathy gets for that, but the tech works.

I almost edited in something about 2018 vs 2026 but didn’t, trusted you to understand :)

Re: Microgpt explained interactively

#49

Post by danhergir »

growingswe wrote: post #32
danhergir wrote: post #25

I went through the article, and it makes sense to me that we're getting names as an output, but why doing so with names?

Names is just a random problem to demonstrate the model. It could be anything, I believe

What if you just use words

Re: Microgpt explained interactively

#50

Post by gwern »

growingswe wrote: post #33
jmkd wrote: post #13

It says its tailored for beginners, but I don't know what kind of beginner can parse multiple paragraphs like this: "How wrong was the prediction? We need a single number that captures "the model thought the correct answer was unlikely." If the model assigns probability 0.9 to the correct next token, the loss is low (0.1). If it assigns probability 0.01, the loss is high (4.6). The formula is − log ⁡ ( � ) −log(p) wh…

I see. The problem with me writing these is even though I'm not an expert, I do have a bit of knowledge on certain things so I'm prone to say things that make sense to me but not to beginners. I'll rethink it

One of the downsides of using an expert LLM to write for you is that they know all that perfectly well, even if you don't, and aren't too bothered by such a chunk. It's like reading any Wikipedia article on mathematics... This is the kind of thing that people are documenting in the LLM-user literature in creating an illusion of expertise (or 'illusion of transparency'). Because the LLM explains it so fluently, you feel like you understand, even though you don't. Hence new phrases like 'cognitive debt' to try to deal with it.

(This is also why people like cramming or lectures rather than quizzing or spaced repetition, because they produce a certain 'illusion of depth' https://gwern.net/doc/psychology/cognitive-bias/illusion-of-... ).

Post reply on HN