Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

gilesthomas.com

Post reply on HN
Print view View original

11–20 of 124 posts

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#11

Post by DeathArrow »

rvnx wrote: post #9
contrast wrote: post #5

Earlier quoted context omitted.

You seem to be talking about a production-grade model rather than building an LLM as an exercise? Or if not, why do you disagree with the article's example of building a small LLM for $100?

I think I should have replied as a totally separate comment. This is my mistake. It is nice that the author shared the results of his exercise / experiment. Just got sad as I was reminded (when the 100 USD were mentioned) that all this game is 90%+ about money and hardware rather than skills. That being said I really like the initiative of the author.

That is true for many kinds of software where you need a big amount of resources. No matter how skilled I am, I cannot build Facebook, Google, Photoshop alone. But a tiny version of it just to learn? Why not!

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#12

Post by meehai »

rvnx wrote: post #9
contrast wrote: post #5

Earlier quoted context omitted.

You seem to be talking about a production-grade model rather than building an LLM as an exercise? Or if not, why do you disagree with the article's example of building a small LLM for $100?

I think I should have replied as a totally separate comment. This is my mistake. It is nice that the author shared the results of his exercise / experiment. Just got sad as I was reminded (when the 100 USD were mentioned) that all this game is 90%+ about money and hardware rather than skills. That being said I really like the initiative of the author.

it's skills first and then money and hardware for scale

A more skilled person that understands all the underlying steps will always be more efficient in scaling up due to knowing where to allocate more.

basically... you always need the skills and the money is the fine tuning.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#13

Post by haolez »

Havoc wrote: post #7

> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level sco…

If you can create this filtering model, you have created Skynet and solved AGI :D

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#14

Post by ACCount37 »

ducktective wrote: post #4

Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?

Research runs on a variety of scales - but "check if this new idea/method/architecture isn't completely dumb on small scale before trying to scale up" is a common enough pattern. And most of those fail on small scale.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#15

Post by ACCount37 »

Havoc wrote: post #7

> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level sco…

Data filtering. Dataset curation. Curriculum learning. All already in use.

It's not sexy, it's not a breakthrough, but it does help.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#16

Post by htrp »

ACCount37 wrote: post #14
ducktective wrote: post #4

Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?

Research runs on a variety of scales - but "check if this new idea/method/architecture isn't completely dumb on small scale before trying to scale up" is a common enough pattern. And most of those fail on small scale.

depressingly enough, things that work on small scale architectures often don't work at larger scales

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#17

Post by embedding-shape »

Havoc wrote: post #7

> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level sco…

Makes me wonder what kind of model we could get if we just trained on Wikidata and similar datasets, but pre-processed to be natural language rather than just triplets of data.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#18

Post by ACCount37 »

htrp wrote: post #16
ACCount37 wrote: post #14

Earlier quoted context omitted.

Research runs on a variety of scales - but "check if this new idea/method/architecture isn't completely dumb on small scale before trying to scale up" is a common enough pattern. And most of those fail on small scale.

depressingly enough, things that work on small scale architectures often don't work at larger scales

Yep, most of what's remaining fails to scale. But it's still a very solid filter.

Sure, there are things that don't work on small scale and then work on large scale. But they're rare, and they sure are going to be expensive to find and validate.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#20

Post by lynndotpy »

ducktective wrote: post #4

Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?

If you're seriously doing deep learning research, it's very very nice to own your own GPU.

For four years of AI PhD research I worked with a 1050Ti on a personal laptop and a 2060 on a personal desktop. You can do a lot of validation and development on consumer GPUs.

That said, the OP does not train an LLM from scratch on a 3090. That would not be feasible

Post reply on HN