Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

youtube.com

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#41

Post by Centigonal »

mattnewton wrote: post #39
ingenieroariel wrote: post #14

Earlier quoted context omitted.

On number 2, even if you are John Carmack you may have trouble getting the right people on the phone. https://twitter.com/id_aa_carmack/status/1305967411749892098... Anyone at Google Cloud out there? It seems I can't get my GPU quota raised to 40 x V100 as an independent researcher. I was told that setting up a website would help, but I would rather not. I can pay the bills...

I gave up and moved to lambdalabs, then they ran out of quota across the board, and now I use a combination of Vast, and Coreweave today.

b-but you work for- why wouldn't they... you know what, never mind.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#42

Post by 1vuio0pswjnm7 »

pottspotts wrote: post #11

This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.

Does one really need Python. What about using Lua.

https://news.ycombinator.com/item?id=7928738

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#43

Post by srajabi »

karpathy wrote: post #6
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

It's cool you're on here and I'm sure I speak for many people in saying I really appreciate your video series and comments like the above!

Thank you!

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#44

Post by green-eclipse »

gorango wrote: post #30
nlowell wrote: post #16

Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.

What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...

> likely saved more lives than most doctors at this point

Care to elaborate on this ridiculous claim?

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#46

Post by joedevon »

sva_ wrote: post #8
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…

How much of the purchase price is a barter for Azure compute-time? That may explain a lot.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#47

Post by gorango »

green-eclipse wrote: post #44
gorango wrote: post #30

Earlier quoted context omitted.

What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...

> likely saved more lives than most doctors at this point Care to elaborate on this ridiculous claim?

What's ridiculous about stats? A Tesla with autopilot or FSD is more than 10x less likely to be in a collision, based on various US agencies stats. Average chance of collision is 1 in 366 for every 1000 miles driven. Something like 1 in 20000 of those result in fatalities. Now, take millions of Teslas equipped with any assisted-driving tech, multiply by miles driven, include the above averages, and divide by ten. Doctors would be jelly.

Perhaps you can elaborate on what strikes you as ridiculous about fairly straightforward stats? Surprised to see you on a ML thread...

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#48

Post by tysam_and »

Centigonal wrote: post #41
mattnewton wrote: post #39

Earlier quoted context omitted.

I gave up and moved to lambdalabs, then they ran out of quota across the board, and now I use a combination of Vast, and Coreweave today.

b-but you work for- why wouldn't they... you know what, never mind.

It is sometimes far more convenient to avoid the paper trail and bureaucracy of having to provision things internally, similarly I'm sure to how renting GPUs online for short periods of time avoids the issue of having to pay for the maintenance and time costs of maintaining them onsite.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#49

Post by hluska »

xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

I use this model to help me reason through it.

Realistically, it takes five steps. At a glance, the five steps are simple. But when you dig in, you realize that people have 15+ years of experience in each of the individual steps. At a glance, their insight into the individual problems will seem way too big to actually use, but as you dig deeper, you’ll find edge case after edge case that uses those insights.

So, it’s really five steps, but very smart people have devoted their lives to figuring out each one of those. We’re lucky to live in a time where we can stand on those shoulders, but it can take quite the leap to get up there in the first place.

And of course, at each point you’ll get some nice rewards. It feels good. It’s exciting. And my inner twelve year old feels like this is why we fell in love with tech in the first place.

So, it is quite hard but the fact it exists means it’s obtainable. If you get into it, I have a lot of respect for you and hope you have a ridiculous amount of fun. On the other hand, if you get into it and just don’t enjoy it, who cares? There are many other interesting fields!!

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#50

Post by ultrasounder »

Just started watching and Andrej is an excellent "thingexplainer". His in-depth knowledge of the underlying atoms/bits comes through. As an extra benefit to watching his lecture using the https://karpathy.ai/zero-to-hero.html is the link to his "discord chat". This is a very active community and Andrej is very active there. So feel free to watch lectures, cry through the assignments and come to the discord with questions. And they will be answered.
Post reply on HN