Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

youtube.com

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#51

Post by cjdoc29 »

gorango wrote: post #47
green-eclipse wrote: post #44

Earlier quoted context omitted.

> likely saved more lives than most doctors at this point Care to elaborate on this ridiculous claim?

What's ridiculous about stats? A Tesla with autopilot or FSD is more than 10x less likely to be in a collision, based on various US agencies stats. Average chance of collision is 1 in 366 for every 1000 miles driven. Something like 1 in 20000 of those result in fatalities. Now, take millions of Teslas equipped with any assisted-driving tech, multiply by miles driven, include the above averages, and divide by ten. Doc…

Can you cite "various US agencies stats"? I say this as someone who used FSD Beta for a full year and Autopilot (with Navigate on Autopilot) for over 3 years now. It would actually help the conversation here to cite it, though I never knew third-parties evaluated Tesla's claims/statistics.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#52

Post by gmadsen »

hluska wrote: post #49
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

I use this model to help me reason through it. Realistically, it takes five steps. At a glance, the five steps are simple. But when you dig in, you realize that people have 15+ years of experience in each of the individual steps. At a glance, their insight into the individual problems will seem way too big to actually use, but as you dig deeper, you’ll find edge case after edge case that uses those insights. So, it’s…

As someone who hasn't really done much deep learning, I've always wondered if the work itself is fullfilling or if it is just the fact that there is absurbly cool outcomes? The math isn't super complex, it seems the majority of the effort is data cleaning and tuning. Is it just a massive labor of love? I also worry that the labor itself doesn't build on itself and becomes obsolete knowledge like a web framework.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#53

Post by somenameforme »

anigbrowl wrote: post #13
karpathy wrote: post #6

Earlier quoted context omitted.

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

Extremely interested in your take on where language/reasoning competency ends and knowledge retrieval begins. OpenAI stuff has succeeded in part because it can synthesize good bullshit* on a huge variety of topics. For many purposes this makes it as good as asking someone in the same room to look something up for you on Wikipedia. But while vast general and somewhat special knowledge is very impressive, comprehension…

This is why it has always seemed to me that the 'chat bot' -> AI pathway has felt quite analogous to the 'chess bot' -> AI pathway. We're constantly trying to replicate things that look like demonstrations of intelligence, but never really bothering with what intelligence is. What I mean is that a man lifting 400kg is a demonstration of exceptional athleticism. A 400kg man sitting on a balance and having 400kg go up on the other side is not, even though if we only observe the output (400kg goes up) then it is absolutely identical.

This isn't just a 'only humans can be intelligent' type argument, but emphasizing that what we want and what we're pursuing seem to be quite different. Newton deriving the inverse square law of gravitational attraction by observing things fall on Earth and watching the celestial bodies in the sky - that is an application of the sort of intelligence that we want. Asking a student to memorize and later recite that the gravitational force is proportional to m1*m2/r^2 is the sort of intelligence that we're building. And it's not like the latter leads to the former, of course it's the exact opposite!

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#54

Post by albert_e »

I am a simple man. I see a video post by karpathy, I upvote and watch.

I discovered Andrej very recently and I am a huge fan. Kudos to this whole effort!

Two ideas --

1. While these explainers are outstanding -- I can think of supplementary material/presentation that can nicely complement these explanations if they are presented visually. Especially the concepts of multidimensional tensors.

Something like what 3B1B (or his followers that create 'Summer of Math Exposition' videos) does -- which is not a skill I have.

I am thinking of creating some visual slides (my forte) but would there be interest in making this a larger collaboration that creates explainers for "visual learners"?

2. There should really be a discussion forum for people who follow along these "make more" tutorials -- to have discussions about each specific video, infact each specific timestamped chapter of these videos in that context.

Is there a framework or tool that lets us integrate YouTube videos and timestamped chapters into a "discussion forum" -- whether a simple website or a discord/slack. Once again this is slightly outside my skillset but if it appeals to people, maybe some ideas and effort can come together to make this happen?

EDIT: #facepalm -- I see there is already a discord [0] on the webpage [1] (but not likely tied to very specific chapters as I imagined -- but should be a excellent start anyway)

[0] - https://discord.gg/3zy8kqD9Cp [1] - https://karpathy.ai/zero-to-hero.html

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#56

Post by eddsh1994 »

karpathy wrote: post #6
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

I would love to know what your thoughts are on how software engineering (and jobs in general) will change over the next 10 years and what we lowly developers can do to keep up & maybe even be involved in that change

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#57

Post by zerd »

javajosh wrote: post #19
pottspotts wrote: post #11

This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.

Same, but one that doesn't assume python or an operating system.

Sounds like a project for Ben Eater https://eater.net/

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#58

Post by eddsh1994 »

gorango wrote: post #47
green-eclipse wrote: post #44

Earlier quoted context omitted.

> likely saved more lives than most doctors at this point Care to elaborate on this ridiculous claim?

What's ridiculous about stats? A Tesla with autopilot or FSD is more than 10x less likely to be in a collision, based on various US agencies stats. Average chance of collision is 1 in 366 for every 1000 miles driven. Something like 1 in 20000 of those result in fatalities. Now, take millions of Teslas equipped with any assisted-driving tech, multiply by miles driven, include the above averages, and divide by ten. Doc…

Millions of Teslas equipped with ‘any assisted-driving tech’ != autopilot on FSD. We have to compare deaths in Teslas with deaths in other vehicles too, as that’s what people would use if not a Tesla.

The chance of fatality per collision is way off.

If you’re going to go down a rabbit hole we need to look at lives impacted by lithium mining compared to regular combustion engine vehicles.

While I appreciate your point I’d be surprised if the number of QALYs is higher from working on FSD at Tesla compared to being a doctor.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#59

Post by tysam_and »

somenameforme wrote: post #53
anigbrowl wrote: post #13

Earlier quoted context omitted.

Extremely interested in your take on where language/reasoning competency ends and knowledge retrieval begins. OpenAI stuff has succeeded in part because it can synthesize good bullshit* on a huge variety of topics. For many purposes this makes it as good as asking someone in the same room to look something up for you on Wikipedia. But while vast general and somewhat special knowledge is very impressive, comprehension…

This is why it has always seemed to me that the 'chat bot' -> AI pathway has felt quite analogous to the 'chess bot' -> AI pathway. We're constantly trying to replicate things that look like demonstrations of intelligence, but never really bothering with what intelligence is. What I mean is that a man lifting 400kg is a demonstration of exceptional athleticism. A 400kg man sitting on a balance and having 400kg go up…

hence the artificial in the name

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#60

Post by andruby »

notjonheyman wrote: post #55

I had the please of working under Andrej a few years ago. It was very interesting the problems we were solving until this other guy that is a billionaire would throw a wrench (and fits) into the whole thing.

I’d be interested to hear more. It was surprising to hear of Andrej leaving.
Post reply on HN