Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Build and train GPT-2 from scratch using PyTorch

differ.blog

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Build and train GPT-2 from scratch using PyTorch

#11

Post by hombre_fatal »

nottorp wrote: post #9
elpocko wrote: post #8

Earlier quoted context omitted.

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

How can a video be better than text based info when we're talking about a programming task, which is all text?

Because the value isn't in just having raw reference material like complex code you just paste into your program. The value is in communicating ideas and understanding.

You don't need to skim through the 4 hour video long to see it's full of whys and hows and explanations and demonstrations that massively dwarf the blog post. It's basically a mini course.

Programming is an interactive process, so transcribing the video to text also removes a lot information. The video isn't just an audio form of what could've been a long text tutorial. You're watching someone do something.

Re: Build and train GPT-2 from scratch using PyTorch

#12

Post by frognumber »

elpocko wrote: post #8
rty32 wrote: post #2

Andrej Karpathy's video is probably much better than this: https://youtu.be/l8pRSuU81PU

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

Convert video to text with this one simply secret trick:

* yt-dlp

* whisper

(whisper is surprisingly good for a lot of educational videos)

Even better is to connect text to images. but that's less of a simple trick (although I do have the code).

Re: Build and train GPT-2 from scratch using PyTorch

#13

Post by throwaway71271 »

elpocko wrote: post #8
rty32 wrote: post #2

Andrej Karpathy's video is probably much better than this: https://youtu.be/l8pRSuU81PU

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

This particular video is an exceptionally awesome video.

And so are his other neural net videos.

He keeps bringing you back to first principles, again and again, does not let a single doubt remain, and the code is very accessible and practical.

The intersection between those who can teach and those who truly understand the deepest essence of a subject is much smaller than we would hope. People like Feynman, Tanenbaum, Sussman, Susskind, and now Karpathy are exceedingly rare. Each of them is a gift for generations to come. So, when you find one whose style resonates with the way you think, I suggest watching their videos multiple times. :)

Re: Build and train GPT-2 from scratch using PyTorch

#14

Post by nottorp »

hombre_fatal wrote: post #11
nottorp wrote: post #9

Earlier quoted context omitted.

How can a video be better than text based info when we're talking about a programming task, which is all text?

Because the value isn't in just having raw reference material like complex code you just paste into your program. The value is in communicating ideas and understanding. You don't need to skim through the 4 hour video long to see it's full of whys and hows and explanations and demonstrations that massively dwarf the blog post. It's basically a mini course. Programming is an interactive process, so transcribing the vid…

> The value is in communicating ideas and understanding.

And those ideas absolutely can't be communicated in concise writing, I have to waste my life on listening to talking heads?

Re: Build and train GPT-2 from scratch using PyTorch

#15

Post by hombre_fatal »

nottorp wrote: post #14
hombre_fatal wrote: post #11

Earlier quoted context omitted.

Because the value isn't in just having raw reference material like complex code you just paste into your program. The value is in communicating ideas and understanding. You don't need to skim through the 4 hour video long to see it's full of whys and hows and explanations and demonstrations that massively dwarf the blog post. It's basically a mini course. Programming is an interactive process, so transcribing the vid…

> The value is in communicating ideas and understanding. And those ideas absolutely can't be communicated in concise writing, I have to waste my life on listening to talking heads?

Yes. Skim the video. You're watching someone explain things interactively in a way that cannot be done with text.

Whether you have the preference for it or not is not up for debate here.

Re: Build and train GPT-2 from scratch using PyTorch

#16

Post by omerhac »

frognumber wrote: post #12
elpocko wrote: post #8

Earlier quoted context omitted.

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

Convert video to text with this one simply secret trick: * yt-dlp * whisper (whisper is surprisingly good for a lot of educational videos) Even better is to connect text to images. but that's less of a simple trick (although I do have the code).

Thats cool but not exactly the same as reading text that was written to be consumed as text. I think a big part of the reason technical text is easier to digest is the way it's written, and not so much the medium.

Re: Build and train GPT-2 from scratch using PyTorch

#18

Post by omerhac »

Cool blog, thanks!

I did a similar project a couple of years ago for a university course, only I also added style transfer, it turned out pretty cool. I scraped a bunch of news data together with it's news section and trained a self attention language model from scratch, turned out pretty hilarious. Data was in Hebrew, which is a challenge to tokenize because of the morphology. I posted it on ArXiV if someone's interested in the style transfer and tokenization process: https://arxiv.org/abs/2212.03019

Re: Build and train GPT-2 from scratch using PyTorch

#19

Post by nottorp »

hombre_fatal wrote: post #15
nottorp wrote: post #14

Earlier quoted context omitted.

> The value is in communicating ideas and understanding. And those ideas absolutely can't be communicated in concise writing, I have to waste my life on listening to talking heads?

Yes. Skim the video. You're watching someone explain things interactively in a way that cannot be done with text. Whether you have the preference for it or not is not up for debate here.

> Skim the video.

So not all 4 hours are worth watching? :)

Re: Build and train GPT-2 from scratch using PyTorch

#20

Post by moffkalast »

That's cool as a learning experience, but if you're gonna build a language transformer, why not instead of ClosedAI's outdated nonsense learn something with a more established open architecture like llama, so whatever you end up training ends up plug and play compatible with every LLM tool in the universe when converted to a GGUF?

Otherwise it's like learning to build a website and stopping short of actually doing the final bit where you put it on a webserver and run it live.

Post reply on HN