Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

Post by AlexandrB » Wed, Jan 18, 2023, 2:40 PM UTC

cjdoc29 wrote: post #51

gorango wrote: post #47

Earlier quoted context omitted.

What's ridiculous about stats? A Tesla with autopilot or FSD is more than 10x less likely to be in a collision, based on various US agencies stats. Average chance of collision is 1 in 366 for every 1000 miles driven. Something like 1 in 20000 of those result in fatalities. Now, take millions of Teslas equipped with any assisted-driving tech, multiply by miles driven, include the above averages, and divide by ten. Doc…

Can you cite "various US agencies stats"? I say this as someone who used FSD Beta for a full year and Autopilot (with Navigate on Autopilot) for over 3 years now. It would actually help the conversation here to cite it, though I never knew third-parties evaluated Tesla's claims/statistics.

Post by naasking » Wed, Jan 18, 2023, 3:11 PM UTC

somenameforme wrote: post #80

naasking wrote: post #76

Earlier quoted context omitted.

We don't know how to define the essence of intelligence, so all we can do is knock all the strawmen we traditionally attribute to intelligence, until we're either left with the essence of true intelligence, or we knock off everything and we find intelligence was just a bunch of tricks after all.

I don't think the essence is especially elusive. It's the ability to make novel, meaningful, and useful discoveries from precepts that don't immediately "obviously" lead to those discoveries. Observing the sky and nature leading to a mathematical formulation of gravity is an absolutely amazing leap. In part because of what was done, but perhaps even more so for even beginning to imagine it was something that could be…

Post by superflit » Wed, Jan 18, 2023, 4:27 PM UTC

karpathy wrote: post #6

xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

Post by gorango » Wed, Jan 18, 2023, 7:23 PM UTC

AlexandrB wrote: post #81

cjdoc29 wrote: post #51

Earlier quoted context omitted.

Can you cite "various US agencies stats"? I say this as someone who used FSD Beta for a full year and Autopilot (with Navigate on Autopilot) for over 3 years now. It would actually help the conversation here to cite it, though I never knew third-parties evaluated Tesla's claims/statistics.

There's also a strong selection bias to a bulk stat like the one parent mentioned. Teslas are expensive cars and I'm guessing their drivers are likely to be older, more affluent, and more educated that average - this all correlates to lower accidents/fatalities regardless of FSD[1][2]. [1] https://pubmed.ncbi.nlm.nih.gov/24103823/ [2] https://personalinjurylawyersaustintx.com/blog/education-lev...

Post by tysam_and » Wed, Jan 18, 2023, 8:08 PM UTC

rockwotj wrote: post #40

karpathy wrote: post #6

Earlier quoted context omitted.

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links The pretrained model is already available.