[deleted]
Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
101–110 of 111 posts
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#102Earlier quoted context omitted.
We don't know how to define the essence of intelligence, so all we can do is knock all the strawmen we traditionally attribute to intelligence, until we're either left with the essence of true intelligence, or we knock off everything and we find intelligence was just a bunch of tricks after all.
I don't think the essence is especially elusive. It's the ability to make novel, meaningful, and useful discoveries from precepts that don't immediately "obviously" lead to those discoveries. Observing the sky and nature leading to a mathematical formulation of gravity is an absolutely amazing leap. In part because of what was done, but perhaps even more so for even beginning to imagine it was something that could be…
> practically infinite storage, infinitely more accurate recall
You can already hold all of humanity's written cultural knowledge on a thumbnail-sized drive. Recall is pretty accurate, too.
> revolutionizing humanity
Tell that anyone who lived 100 or 1000 years ago. World-wide instant communication. A cultural species of interconnected thought, a network the size of a planet. Augmented with tools of perfect memory recall, error-free precision calculations, way beyond what an unaugmented human brain can do. Welcome to the present. We even got remote-controlled robots working for us on other planets.
I'll keep my ad-blocker enabled, though. But allow me to think of it as a brain augmentation. We do what we always did: cultural evolution. We are creating tools to augment ourselves, not machines that are independent from us. As we integrate those tools into our daily routine, the tools are shaping our practice and needs. So we again create different tools, or ways of living. We diversify, then copy the successful. What's the point of creating a new god of intelligence? We have plenty of gods already. Let's maybe study and discuss non-human intelligence instead. (Edit: rewrote last paragraph.)
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#103Earlier quoted context omitted.
I would define intelligence not in terms of search, but creation. And the two are indeed different. As one simple example - early man had no concept of math, or even numbers. Incidentally, the same is even true of some isolated tribes to this day [1]. Somehow we created numbers, seemingly from nothing. And it was only this creation that enabled us to move onto even more creation where the search space continues to gr…
But everything expressible by humans is expressible by a Turing machine, so there is no fundamental difference between search and creation since Turing machines are recursively enumerable. We didn't create numbers seemingly from nothing, it was necessary to track our food and our children or family. Even crows have the ability to count.
Again, read the paper. Numbers were thought to be an intuitive concept - they are not, not even amongst humans. One needs not numbers to keep track of their family or food anymore than they need calculus to say, "Wow that thing's speeding up." They have "one", "two", and "many" and that works for all their purposes.
Basically look to any example, where what is discovered is not a recombination of preexisting knowledge but the emergence of new knowledge and you'll find search is pointless. As an example, consider hand washing. Now a days we all intuit that hand washing is a good way to prevent the spread of disease. But of course that intuition is because we all know of and accept a germ based theory of disease. A couple of hundred years ago this was not true. Go a little further back and the concept of germs did not even exist. And so surgeons did not regularly wash their hands even before doing things like surgery.
Now I challenge you to, even in wildly hand-wavey fashion, to describe the creation of a turing machine that could, from the basis of knowledge of an individual of such times, "discover" the secret of hand washing. There were no records kept on hand washing : illness rates or anything of the sort, because nobody even stopped to consider the impact it might be having.
The difficulty you're going to face here is that there is no preexisting knowledge to draw upon. You are not "searching" for an answer, but having to create it, seemingly from nothing.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#104Earlier quoted context omitted.
But everything expressible by humans is expressible by a Turing machine, so there is no fundamental difference between search and creation since Turing machines are recursively enumerable. We didn't create numbers seemingly from nothing, it was necessary to track our food and our children or family. Even crows have the ability to count.
Again, read the paper. Numbers were thought to be an intuitive concept - they are not, not even amongst humans. One needs not numbers to keep track of their family or food anymore than they need calculus to say, "Wow that thing's speeding up." They have "one", "two", and "many" and that works for all their purposes. Basically look to any example, where what is discovered is not a recombination of preexisting knowledg…
> Again, read the paper. Numbers were thought to be an intuitive concept - they are not, not even amongst humans.
Your link doesn't prove anything, it contains testimony from experts arguing both sides. Odd that you think one side is automatically correct from one study that was inconclusive and a clear example of an exception to the rule, at best.
> Basically look to any example, where what is discovered is not a recombination of preexisting knowledge but the emergence of new knowledge and you'll find search is pointless
I think you'll find it much harder to argue this point than you think. Most such discoveries result from simple observations of the world, so the information was already out there, people just didn't notice it before.
> Now I challenge you to, even in wildly hand-wavey fashion, to describe the creation of a turing machine that could, from the basis of knowledge of an individual of such times, "discover" the secret of hand washing
Exactly the way it happened: someone noticed that fewer people died in hospitals where the doctors washed their hands after performing autopsies. The scientific process is reliable because it's mechanistic, repeatable. All scientific knowledge derives from simple, repeatable observations like this.
The closest thing you'll find to true invention is maybe math and various logics. But even then, this is often simply a process of permuting existing axioms, and adding a new randomly generated axiom to see if anything interesting haopens. This is a search process, most of whose results will be internally inconsistent and so get discarded quickly by the human mind with it's effective pattern matching.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#105Earlier quoted context omitted.
Again, read the paper. Numbers were thought to be an intuitive concept - they are not, not even amongst humans. One needs not numbers to keep track of their family or food anymore than they need calculus to say, "Wow that thing's speeding up." They have "one", "two", and "many" and that works for all their purposes. Basically look to any example, where what is discovered is not a recombination of preexisting knowledg…
> Again, read the paper. Numbers were thought to be an intuitive concept - they are not, not even amongst humans. Your link doesn't prove anything, it contains testimony from experts arguing both sides. Odd that you think one side is automatically correct from one study that was inconclusive and a clear example of an exception to the rule, at best. > Basically look to any example, where what is discovered is not a re…
Hand washing records were not kept for the same reason I've mentioned multiple times - nobody ever thought it relevant, so it wasn't considered relevant. So it's not like you can simply search the records for something which does not exist! Even when one doctor finally did discover the value of handwashing, through a remarkable degree of serendipity, his hypothesis, based on anecdotal evidence, was rejected because it did not line up with scientific thought of the time. He ended up in an insane asylum and died. Like a Greek tragedy, his cause of death was an infected wound on his hand, very possibly caused by excessive washing! [1]
So now we return to the same question. How do you expect a machine to simply discover the value of hand washing? Let alone carry out tests? You have no data on hand washing whatsoever as it's not seen as relevant. It's a rather random hypothesis that, given the knowledge of the time, would have less than zero basis for support. There is no logical reason for its discovery, nor ought it ever be prioritized highly in any way whatsoever.
And this is, in many ways, the rule more than the exception for discovery.
[1] - https://www.nationalgeographic.com/history/article/handwashi...
----
On the numbers issue. I was not referring to "views" but facts. The tribesmen had terms only for 1, 2, and many. And while the article doesn't mention it, it's safe to assume they have 0 systems of mathematics. Assuming they are not uniquely retarded, we were all in a similarly limited state of understanding at some point. Searching for where we are, from the state of where they are, will yield no results. Yet somehow, we achieved it.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#106Earlier quoted context omitted.
I was so confused by the saltiness until I saw the username. I'm sure you've earned it. I got into deep learning because of your char-rnn posts a while ago -- it inspired me to do an undergrad thesis on the topic. I read arxiv papers after that and implemented things from the ground up until a startup liked my work and hired me in a neural network engineer position. Fast forward a few years and I was enamoured with m…
> I was so confused by the saltiness I was confused by what you thought was salty. I don't see it remotely.
Same. I think it just lacks wide-eyed optimism.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#107I really hate the phrase "from scratch" in the title.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#108I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
Thanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following:
> rough steps:
> 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $.
Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ?
> 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts.
It seems like the standard instructGPT model itself is based on a 1 billion param GPT model. Wouldn't that fit on a 24GB RTX 3090 ? Might take longer, maybe not enough opportunity for hyper-parameter search, but still possible right ? Or is hyper-parameter search on a thousand machines in parallel the real magic sauce here ?
> 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec...
Sounds like a good opportunity to learn. No pain, no gain :-)
> 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
Maybe somebody would open source the equivalent datasets for this soon ? Otherwise the data collection seems prohibitively expensive for somebody trying to do this for fun: contract expert annotators, train them, annotate/reannotate for months ?
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#109Earlier quoted context omitted.
b-but you work for- why wouldn't they... you know what, never mind.
Oh, I quit recently. Very surprisingly, I learned it was harder to get access to GPUs at big tech companies outside their dedicated research teams than it was for a scrappy hacker outside on side projects with lots of savings. So I quit to work on those projects. I miss not having a dedicated infra team but don’t miss having to beg for resources. I wanted to use google cloud since I use some of their other services f…
Out of curiosity, just how much resources are you able to access with google colab and can you use it for AI training? I’ve been becoming more interested in fintech and trying my hand at predictive/pattern recognition AI but the costs are incredibly prohibitive to getting started, both hardware and data.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#110Earlier quoted context omitted.
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
On number 2, even if you are John Carmack you may have trouble getting the right people on the phone. https://twitter.com/id_aa_carmack/status/1305967411749892098... Anyone at Google Cloud out there? It seems I can't get my GPU quota raised to 40 x V100 as an independent researcher. I was told that setting up a website would help, but I would rather not. I can pay the bills...
@ingenieroariel, sorry for this trouble.
I am product manager for Cloud TPU, I would be happy to connect you with my GPU colleagues and also explore if Cloud TPU can help with your research as well.
What's the best way to connect with you?