Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt explained interactively

growingswe.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Microgpt explained interactively

#31

Post by growingswe »

davidw wrote: post #14
malnourish wrote: post #11

I read through this entire article. There was some value in it, but I found it to be very "draw the rest of the owl". It read like introductions to conceptual elements or even proper segues had been edited out. That said, I appreciated the interactive components.

It started off nicely but before long you get "The MLP (multilayer perceptron) is a two-layer feed-forward network: project up to 64 dimensions, apply ReLU (zero out negatives), project back to 16" Which starts to feel pretty owly indeed. I think the whole thing could be expanded to cover some more of it in greater depth.

I tried to include tooltips in some places that go into more depth, but I understand there's a jump. I'm not sure what will be the best way to go about it tbh

Re: Microgpt explained interactively

#33

Post by growingswe »

jmkd wrote: post #13

It says its tailored for beginners, but I don't know what kind of beginner can parse multiple paragraphs like this: "How wrong was the prediction? We need a single number that captures "the model thought the correct answer was unlikely." If the model assigns probability 0.9 to the correct next token, the loss is low (0.1). If it assigns probability 0.01, the loss is high (4.6). The formula is − log ⁡ ( � ) −log(p) wh…

I see. The problem with me writing these is even though I'm not an expert, I do have a bit of knowledge on certain things so I'm prone to say things that make sense to me but not to beginners. I'll rethink it

Re: Microgpt explained interactively

#34

Post by refulgentis »

tibbar wrote: post #28
refulgentis wrote: post #22

Earlier quoted context omitted.

People aren't gonna be happy I spell this out, but, Karpathy's not The Dude. He's got a big Twitter following so people assume somethings going on or important, but he just isn't. Biggest thing he did in his career was feed Elon's Full Self Driving delusion for years and years and years. Note, then, how long he lasted at OpenAI, and how much time he spends on code golf. If you're angry to read this, please, take a mi…

I have no skin in the game here, but this seems a bit "sharp-edged", do you have something against the guy? He just seems deep into his influencer/retired hobbyist arc to me...

No, and me too. Just had been sitting in my chest a while when I see people expecting non-hobbyist work from him. And had been worried to post it because things you and I understand become sharp-edged when spoken out loud to other people who don't.

Re: Microgpt explained interactively

#35

Post by thebiblelover7 »

I know many comments mentioned that it was too introductory, or too deep. But as someone that does not have much experience understanding how these models work, I found this overview to be pretty great.

There were some concepts I didn't quite understand but I think this is a good starting point to learning more about the topic.

Re: Microgpt explained interactively

#36

Post by UltraSane »

growingswe wrote: post #29
jsheard wrote: post #9

Earlier quoted context omitted.

The rate they are posting new articles on random subjects is also a pretty indicative of a content mill. In 3 days they've covered machine learning, geometry, cryptography, file formats and directory services.

I had to look up what a content mill is. I'm not one, I think. It's "random" stuff because my interests are different. These posts are not written sequentially, I've been working on them (except for this MicroGPT one) for weeks and only publishing now.

It has gotten to the point you need timestamped keystrokes and a screen recording to prove you actually wrote something yourself.

Re: Microgpt explained interactively

#37

Post by bonoboTP »

UltraSane wrote: post #36
growingswe wrote: post #29

Earlier quoted context omitted.

I had to look up what a content mill is. I'm not one, I think. It's "random" stuff because my interests are different. These posts are not written sequentially, I've been working on them (except for this MicroGPT one) for weeks and only publishing now.

It has gotten to the point you need timestamped keystrokes and a screen recording to prove you actually wrote something yourself.

Soon you'll be able to generate a video with AI that shows you typing the entire thing, and will narrate it in your own voice with voice cloning.

Re: Microgpt explained interactively

#39

Post by malnourish »

growingswe wrote: post #31
davidw wrote: post #14

Earlier quoted context omitted.

It started off nicely but before long you get "The MLP (multilayer perceptron) is a two-layer feed-forward network: project up to 64 dimensions, apply ReLU (zero out negatives), project back to 16" Which starts to feel pretty owly indeed. I think the whole thing could be expanded to cover some more of it in greater depth.

I tried to include tooltips in some places that go into more depth, but I understand there's a jump. I'm not sure what will be the best way to go about it tbh

I liked the tooltips. You should define each term the first time it shows up (MLP for example).

Re: Microgpt explained interactively

#40

Post by gwern »

growingswe wrote: post #29
jsheard wrote: post #9

Earlier quoted context omitted.

The rate they are posting new articles on random subjects is also a pretty indicative of a content mill. In 3 days they've covered machine learning, geometry, cryptography, file formats and directory services.

I had to look up what a content mill is. I'm not one, I think. It's "random" stuff because my interests are different. These posts are not written sequentially, I've been working on them (except for this MicroGPT one) for weeks and only publishing now.

Dude, you literally start the article with

> Andrej Karpathy wrote a 200-line Python script that trains and runs a GPT from scratch, with no libraries or dependencies, just pure Python.

Almost immediately afterwards, you have a section titled "Numbers, not letters". Need I go on?

Interestingly, despite all the AI tics, the opening passes Pangram as 100% human... though all the following sections I randomly checked also come back as 100% AI. So the simplest explanation would be that you are operating adversarially and you tweaked the opening to target Pangram (perhaps through a anti-AI-detection service, which now exist and are being used by the cutting edge, as Pangram is known to be relatively easy to beat, similar to how people started search-and-replacing em-dashes when that got a little too well known), which unfortunately means I now expect you to lie to me in your response since you apparently went that far to start building up clout.

(BTW, how did you accidentally pick 4 rare names which were in the dataset? "Thanks, will fix" is not a real response to that observation. Are you also going to remove all of the 'just pure X' and 'Y, not X' constructions from your posts now that I've pointed it out?)

Post reply on HN