Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

youtube.com

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#91

Post by Art9681 »

belltaco wrote: post #78
lvl102 wrote: post #69

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

I am not getting the angle here. Anyone, including you, can write GPT-like code, train the model with public data and release it for free. It may cost a few million to train in GPU costs, but if what you say is that important, surely there are folks here(I am assuming a good chunk of HN folks have a decent amount of disposable income) who will donate it for the public good? If there are not, then they either don't co…

I'm interested in solving problems you mention in this space. For the sake of simplicity, I will also agree that the data and the models are free if you know how and where to look. The problem is what then? Who has the money and/or compute capacity to do the work at a scale that can compete with the industry behemoths?

I've been slowly building out a home lab to test mesh computing in this space. Perhaps there is a way to carve the workloads into chunks that can be deployed to a distributed mesh of trusted nodes that have a hardware specs suitable for the task. Then somehow aggregated the results and distributes the entire package back to the network of contributors of that compute capacity. In other words, I will agree to lend you my compute capacity in exchange for a copy of the model you are training. I'd love to collaborate with folks and grow this idea and get a legit open source project going.

Let's build the "Constellation". If anyone wants to geek out and make this happen i'd love to chat. art.aquino at compute dot tech

Building compute clusters and cool software is a passion of mine. So i'm looking to build a network of like minded folks without any commitment and just to help each other.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#92

Post by gorango »

eddsh1994 wrote: post #58
gorango wrote: post #47

Earlier quoted context omitted.

What's ridiculous about stats? A Tesla with autopilot or FSD is more than 10x less likely to be in a collision, based on various US agencies stats. Average chance of collision is 1 in 366 for every 1000 miles driven. Something like 1 in 20000 of those result in fatalities. Now, take millions of Teslas equipped with any assisted-driving tech, multiply by miles driven, include the above averages, and divide by ten. Doc…

Millions of Teslas equipped with ‘any assisted-driving tech’ != autopilot on FSD. We have to compare deaths in Teslas with deaths in other vehicles too, as that’s what people would use if not a Tesla. The chance of fatality per collision is way off. If you’re going to go down a rabbit hole we need to look at lives impacted by lithium mining compared to regular combustion engine vehicles. While I appreciate your point…

You're probably right that fatalities per collision is off but plug in your own numbers and then divide by ten... the point still stands. Autopilot has saved countless lives. And it was architected by a very small team, led by this one man. But I'll admit, the comparison with doctors is a little hyperbolic. Although in a few years of growth and global scale, who knows...?

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#93

Post by somenameforme »

naasking wrote: post #83
somenameforme wrote: post #80

Earlier quoted context omitted.

I don't think the essence is especially elusive. It's the ability to make novel, meaningful, and useful discoveries from precepts that don't immediately "obviously" lead to those discoveries. Observing the sky and nature leading to a mathematical formulation of gravity is an absolutely amazing leap. In part because of what was done, but perhaps even more so for even beginning to imagine it was something that could be…

> I don't think the essence is especially elusive. It's the ability to make novel, meaningful, and useful discoveries from precepts that don't immediately "obviously" lead to those discoveries. But that's trivial simply by enumerating all Turing machines that reproduces the observed outputs. The point of intelligence is that it somehow "filters" the space of possible theories in some specific, computable way. The ide…

I would define intelligence not in terms of search, but creation. And the two are indeed different. As one simple example - early man had no concept of math, or even numbers. Incidentally, the same is even true of some isolated tribes to this day [1]. Somehow we created numbers, seemingly from nothing. And it was only this creation that enabled us to move onto even more creation where the search space continues to grow ever wider, yet we continue to pull something from nothing. It's not like we had any real basis for the formulation of numbers, or even reason to imagine they existed.

The further you go back in our development, the greater the distinction between creation and search becomes. The article itself even gets into a bit of a paradox on this note. It suggests that language defines thought, and since they have no numbers in their language - they cannot think about numbers. But then how do we have numbers? Somebody was certainly able to, and it's not because they started with numbers in their language. And for that matter how do we even have language? Another thing that was developed from absolutely nothing. Go for enough back in our evolutionary timeline and we wouldn't have even had the ability to express e.g. 'angry noise'. Yet somehow, we created such things - again seemingly from nothing. And I think that is the purest essence of intelligence.

[1] - https://www.nature.com/articles/news040816-10

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#94

Post by trosi »

bitL wrote: post #26
tysam_and wrote: post #20

Earlier quoted context omitted.

Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.

Abacus-driven deep learning with trained monkeys flipping gradients.

Reminds of that part in the Three-Body Problem novel where the emperor of the videogame builds a CPU with soldiers as bits

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#95

Post by sandkoan »

Art9681 wrote: post #91
belltaco wrote: post #78

Earlier quoted context omitted.

I am not getting the angle here. Anyone, including you, can write GPT-like code, train the model with public data and release it for free. It may cost a few million to train in GPU costs, but if what you say is that important, surely there are folks here(I am assuming a good chunk of HN folks have a decent amount of disposable income) who will donate it for the public good? If there are not, then they either don't co…

I'm interested in solving problems you mention in this space. For the sake of simplicity, I will also agree that the data and the models are free if you know how and where to look. The problem is what then? Who has the money and/or compute capacity to do the work at a scale that can compete with the industry behemoths? I've been slowly building out a home lab to test mesh computing in this space. Perhaps there is a w…

For a distributed computing/BitTorrent-style method of running these LLMs, see: https://github.com/bigscience-workshop/petals.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#96

Post by naasking »

somenameforme wrote: post #93
naasking wrote: post #83

Earlier quoted context omitted.

> I don't think the essence is especially elusive. It's the ability to make novel, meaningful, and useful discoveries from precepts that don't immediately "obviously" lead to those discoveries. But that's trivial simply by enumerating all Turing machines that reproduces the observed outputs. The point of intelligence is that it somehow "filters" the space of possible theories in some specific, computable way. The ide…

I would define intelligence not in terms of search, but creation. And the two are indeed different. As one simple example - early man had no concept of math, or even numbers. Incidentally, the same is even true of some isolated tribes to this day [1]. Somehow we created numbers, seemingly from nothing. And it was only this creation that enabled us to move onto even more creation where the search space continues to gr…

But everything expressible by humans is expressible by a Turing machine, so there is no fundamental difference between search and creation since Turing machines are recursively enumerable.

We didn't create numbers seemingly from nothing, it was necessary to track our food and our children or family. Even crows have the ability to count.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#97

Post by poulpy123 »

sva_ wrote: post #8
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…

By single GPU, is a normal one would suffice?

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#98

Post by mycall »

karpathy wrote: post #6
xt00 wrote: post #5

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…

GPUs are much less efficient than cores of the type Cerebras has made.

Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

#99

Post by mensetmanusman »

nlowell wrote: post #16

Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.

Tesla kick started the global EV revolution, so probably favorably regardless of the few negatives that inherently come along the way.
Post reply on HN