LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
1–10 of 124 posts
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#2I think this is a very valuable exercise if you try to understand how LLMs work and if you have the time.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#3I think this is a very valuable exercise if you try to understand how LLMs work and if you have the time.
Sadly to go beyond an exercise, having the money is really what you need if you actually want LLMs now, not time.
Nowadays training very powerful LLMs is easy because all the tooling, source-codes, training datasets, and teaching agents are available.
Getting access to dozens of millions of USD or more is not easy, and for big players this is a just drop in their ocean.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#4Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#5I think this is a very valuable exercise if you try to understand how LLMs work and if you have the time.
Sadly to go beyond an exercise, having the money is really what you need if you actually want LLMs now, not time. Nowadays training very powerful LLMs is easy because all the tooling, source-codes, training datasets, and teaching agents are available. Getting access to dozens of millions of USD or more is not easy, and for big players this is a just drop in their ocean.
You seem to be talking about a production-grade model rather than building an LLM as an exercise? Or if not, why do you disagree with the article's example of building a small LLM for $100?
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#6I think this is a very valuable exercise if you try to understand how LLMs work and if you have the time.
Sadly to go beyond an exercise, having the money is really what you need if you actually want LLMs now, not time. Nowadays training very powerful LLMs is easy because all the tooling, source-codes, training datasets, and teaching agents are available. Getting access to dozens of millions of USD or more is not easy, and for big players this is a just drop in their ocean.
[dead]
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#7> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet
Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level score. Or even a tiny model trained for the task to pick out good data?
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#8Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?
It depends on what you want to do in this gigantic field.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#9Earlier quoted context omitted.
Sadly to go beyond an exercise, having the money is really what you need if you actually want LLMs now, not time. Nowadays training very powerful LLMs is easy because all the tooling, source-codes, training datasets, and teaching agents are available. Getting access to dozens of millions of USD or more is not easy, and for big players this is a just drop in their ocean.
You seem to be talking about a production-grade model rather than building an LLM as an exercise? Or if not, why do you disagree with the article's example of building a small LLM for $100?
I think I should have replied as a totally separate comment. This is my mistake.
It is nice that the author shared the results of his exercise / experiment. Just got sad as I was reminded (when the 100 USD were mentioned) that all this game is 90%+ about money and hardware rather than skills.
That being said I really like the initiative of the author.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#10I really like this article. I hadn't thought that an RTX 3090 would be capable of generating a sort-of decent small LLM from scratch in a reasonable time, but he shows how in detail.