Build and train GPT-2 from scratch using PyTorch

#1

Post by thunderbong » Sat, Jul 06, 2024, 4:03 AM UTC

Build and train GPT-2 from scratch using PyTorch
differ.blog

Re: Build and train GPT-2 from scratch using PyTorch

#2

Post by rty32 » Sat, Jul 06, 2024, 5:19 AM UTC

Andrej Karpathy's video is probably much better than this:

https://youtu.be/l8pRSuU81PU

Re: Build and train GPT-2 from scratch using PyTorch

#3

Post by [deleted] » Sat, Jul 06, 2024, 6:11 AM UTC

rty32 wrote: post #2

Andrej Karpathy's video is probably much better than this: https://youtu.be/l8pRSuU81PU

[deleted]

Re: Build and train GPT-2 from scratch using PyTorch

#4

Post by KTibow » Sat, Jul 06, 2024, 6:32 AM UTC

Reminds me of TinyStories. I wonder if this architecture is better or worse than the ones it tested.

Re: Build and train GPT-2 from scratch using PyTorch

#5

Post by cjtrowbridge » Sat, Jul 06, 2024, 6:47 AM UTC

Also check out Andrej's new llm.c library which includes a script to do this from scratch with fineweb.

Re: Build and train GPT-2 from scratch using PyTorch

#6

Post by [deleted] » Sat, Jul 06, 2024, 8:18 AM UTC

rty32 wrote: post #2

Andrej Karpathy's video is probably much better than this: https://youtu.be/l8pRSuU81PU

[deleted]

Re: Build and train GPT-2 from scratch using PyTorch

#7

Post by aziis98 » Sat, Jul 06, 2024, 8:31 AM UTC

The link to the repo looks broken

https://github.com/ajeetkharel/gpt2-from-scratch/

Re: Build and train GPT-2 from scratch using PyTorch

#8

Post by elpocko » Sat, Jul 06, 2024, 9:47 AM UTC

rty32 wrote: post #2

Andrej Karpathy's video is probably much better than this: https://youtu.be/l8pRSuU81PU

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

Re: Build and train GPT-2 from scratch using PyTorch

#9

Post by nottorp » Sat, Jul 06, 2024, 10:05 AM UTC

elpocko wrote: post #8

rty32 wrote: post #2

Andrej Karpathy's video is probably much better than this: https://youtu.be/l8pRSuU81PU

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

How can a video be better than text based info when we're talking about a programming task, which is all text?

Re: Build and train GPT-2 from scratch using PyTorch

#10

Post by graovic » Sat, Jul 06, 2024, 10:08 AM UTC

nottorp wrote: post #9

elpocko wrote: post #8

Earlier quoted context omitted.

It's gotta be an exceptionally awesome video to beat written text though. I'll take mediocre text over good video any day.

How can a video be better than text based info when we're talking about a programming task, which is all text?

You can listen someone thought process, that is extremely valuable