Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

gilesthomas.com

Post reply on HN
Print view View original

21–30 of 124 posts

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#21

Post by azath92 »

Havoc wrote: post #7

> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level sco…

For small models this is for sure the way forward, there are some great small datasets out there (check out the tiny stories dataset that limits vocab to a certain age but keeps core reasoning inherent in even simple language https://huggingface.co/datasets/roneneldan/TinyStories https://arxiv.org/abs/2305.07759)

I have less concrete examples but my understanding is that dataset curation is for sure the way many improvements are gained at any model size. Unless you are building a frontier model, you can use a better model to help curate or generate that dataset for sure. TinyStories was generated with GPT-4 for example.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#22

Post by ipnon »

ducktective wrote: post #4

Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?

It's good to have a local GPU. That's like your dev environment. Prod is much more expensive in AI programming than in web programming. So you want to make sure everything is working before you push!

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#23

Post by ineedasername »

ducktective wrote: post #4

Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?

Absolutely. Your model selection has limits of course: best practice for some types of replicable research would be to to use unquantized models, but that still leaves room for smaller Gemma and Llama models.

I’m on a 4080 for a lot of work and it gets well over 50 tokens per second on inference for pretty much anything that fits in VRAM. It’s comparable to a 3090 in compute, the 3090 has 50% more vram, the 4080 has better chip-level support for certain primitives, but that actually matters slightly less using unquantized models, making the 3090 a great choice. The 4080 is better if you want more throuput on inference and use certain common quantize levels.

Training LoRa and fine tunes is highly doable. Yesterday’s project for me, as an example, was training trigger functionality into a single token unused in the vocabulary. Under 100 training examples in the data set, 10 to 50 epochs, extremely usable “magic token” results in under a few minutes at most. This is just an example.

If you look at the wealth of daily entries on arxiv in cs.ai many are using established smaller models with understood characteristics, which makes it easier to understand the result of anything you might do both in your research and in others’ being able to put your results in context.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#24

Post by victorbjorklund »

rvnx wrote: post #3
DeathArrow wrote: post #2

I think this is a very valuable exercise if you try to understand how LLMs work and if you have the time.

Sadly to go beyond an exercise, having the money is really what you need if you actually want LLMs now, not time. Nowadays training very powerful LLMs is easy because all the tooling, source-codes, training datasets, and teaching agents are available. Getting access to dozens of millions of USD or more is not easy, and for big players this is a just drop in their ocean.

Totally. While the LLM:s today are amazing it is a bit sad that you can’t build SOTA models on your own (vs a few years ago where someone with the skills and access to a dataset could build a state of art models)

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#25

Post by victorbjorklund »

DeathArrow wrote: post #11
rvnx wrote: post #9

Earlier quoted context omitted.

I think I should have replied as a totally separate comment. This is my mistake. It is nice that the author shared the results of his exercise / experiment. Just got sad as I was reminded (when the 100 USD were mentioned) that all this game is 90%+ about money and hardware rather than skills. That being said I really like the initiative of the author.

That is true for many kinds of software where you need a big amount of resources. No matter how skilled I am, I cannot build Facebook, Google, Photoshop alone. But a tiny version of it just to learn? Why not!

You could 100% build Facebook. You don’t need any hardcore hardware before you have many users.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#26

Post by joefourier »

lynndotpy wrote: post #20
ducktective wrote: post #4

Are off-shelf GPUs (like one 3090) suitable for modern academic research on current AI advancements or is it better to rent some cloud compute?

If you're seriously doing deep learning research, it's very very nice to own your own GPU. For four years of AI PhD research I worked with a 1050Ti on a personal laptop and a 2060 on a personal desktop. You can do a lot of validation and development on consumer GPUs. That said, the OP does not train an LLM from scratch on a 3090. That would not be feasible

M? The OP literally did train an LLM from scratch in a 3090 (except for the tokenizer), that’s what the whole post is about.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#28

Post by jadbox »

nullbound wrote: post #27

I love the level of detail ( probably, because I see it less and less these days ). It genuinely makes me wonder if anyone tried training LLMs on their own writings ( assuming those bigger than 100+ pages ) and what the results were.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#29

Post by trial3 »

jadbox wrote: post #28
nullbound wrote: post #27

I love the level of detail ( probably, because I see it less and less these days ). It genuinely makes me wonder if anyone tried training LLMs on their own writings ( assuming those bigger than 100+ pages ) and what the results were.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

> These things are now more important than ever

oh definitely. i agree here. can't wait to read the rest of the sentence, probably saying something meaningful about the creative benefits of unstructured writing, or the importance of relying on your own thoughts and language and unique voice in the era of LLMs

> as they can literally help fine-tune agents to help assist you using your personal style.

oh

Post reply on HN