Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

gilesthomas.com

Post reply on HN
Print view View original

101–110 of 124 posts

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#101

Post by djmips »

SecretDreams wrote: post #31
jadbox wrote: post #28

Earlier quoted context omitted.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

Is this what tool and die makers used to feel when going to LOC to train their replacements? Personally, I do not want my likeness to persist after my death, nor do I wish for a company to be able to leverage my likeness after I leave said company.

from context I figure you meant China and/or other places that would take over American manufacturing but I'm curious what LOC means - typo?

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#102

Post by djmips »

Chabsff wrote: post #36
victorbjorklund wrote: post #24

Earlier quoted context omitted.

Totally. While the LLM:s today are amazing it is a bit sad that you can’t build SOTA models on your own (vs a few years ago where someone with the skills and access to a dataset could build a state of art models)

In the grand scheme of things, we've only had about a quarter century where you needed a *very* specific kind of problem where prosumer hardware wasn't adequate across computer science as a whole. It's kind of amazing we got that at all for a while.

If you discard the early days of gigantic expensive computers. I guess it's come full circle after a fashion.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#103

Post by QuadmasterXLII »

BubbleRings wrote: post #35

> …reused its embedding matrix as the weights for the linear layer that projects the context vectors from the last Transformers layer into vocab space to get the logits. At first glance this claim sounds airtight, but it quietly collapses under its own techno-mythology. The so-called “reuse” of the embedding matrix assumes a fixed semantic congruence between representational space and output projection, an assumption…

The second paragraph is highly derivative of the adversarial turbo encabulator, which Schmithuber invented in the 90s. No citation of course.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#104

Post by jadbox »

trial3 wrote: post #29
jadbox wrote: post #28

Earlier quoted context omitted.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

> These things are now more important than ever oh definitely. i agree here. can't wait to read the rest of the sentence, probably saying something meaningful about the creative benefits of unstructured writing, or the importance of relying on your own thoughts and language and unique voice in the era of LLMs > as they can literally help fine-tune agents to help assist you using your personal style. oh

I get it. Both things can be true. Unstructured writing can help you develop as a person. It can also teach your own model the 'real raw human train of thoughts' of your personal journey. Personally I love the idea of booting up great-great-grandpa-model that'll have been trained on his 40 years of almost daily journaling. We are not trying to 'remake him' to be clear- we are talking about being have to have an interaction chat with his personality-vibe as it was recorded by his own hand and in his own words.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#105

Post by futuraperdita »

itissid wrote: post #76
jadbox wrote: post #28

Earlier quoted context omitted.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

I have always wondered if I should be recording all my conversations privately — with consent —with family and friends and then train an LLM to let anyone speak to someone that sounds "like me" when I am gone. I suppose one could order all the data over time -— decades — and then train a model incrementally every decade and imitate me better at a point in time. I suppose one could also narrate thoughts and feelings a…

You've created a text-based version of a Black Mirror episode: https://en.wikipedia.org/wiki/Be_Right_Back

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#106

Post by goosers »

nico wrote: post #53

Has anyone done something like this but with apple silicon instead of a graphics card? Training a small LLM on an M2-M5?

I’m experimenting with this, but using the CPU not the GPU. I’m finishing up writing the series now, but focused more on understanding the architecture than trying to build a useful model. Mine requires talking in the language of Shakespeare, and getting replies in the same, a proof of concept more than a useful tool. https://www.tag1.com/white-paper/part1-tokenization-building...

I was interested in focusing on repeatability and using text sources anyone can legally obtain. It’s been fascinating, but after much experimentation it’s clear that working with more text and more diverse text would be extremely helpful.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#107

Post by ACCount37 »

famouswaffles wrote: post #99
ACCount37 wrote: post #15

Earlier quoted context omitted.

Data filtering. Dataset curation. Curriculum learning. All already in use. It's not sexy, it's not a breakthrough, but it does help.

Curriculum learning is not really a thing for these large SOTA LLM training runs (specifically pre-training). We know it would help, but ordering trillions of tokens of data in this way would be a herculean task.

I've heard things about pre-training optimization. "Soft start" and such. So I struggle to believe that curriculum learning is not a thing on any frontier runs.

Sure, it's a lot of data to sift through, and the time and cost to do so can be substantial. But if you are already planning on funneling all of that through a 1T LLM? You might as well pass the fragments through a small classifier before you do that.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#108

Post by spi »

gpjt wrote: post #92
gpjt wrote: post #52

Earlier quoted context omitted.

OP here -- thanks! I'm in the process of doing some trains using the same code plus DDP on big Lambda Labs machines, and (within the bounds of what I can afford) will hopefully have some interesting results about all of those shortly.

OK, early indicators support both you and Gemini quite strongly re: batch size. On my (somewhat ad-hoc) test dataset, I get losses like this: * OpenAI medium weights: 3.231 * OpenAI small weights: 3.500 * My locally trained model, FineWeb Chinchilla, batch size 6: 3.944 * My locally trained model, FineWeb-Edu Chinchilla, batch size 6: 4.167 * My locally trained model, FineWeb-Edu double Chinchilla, batch size 6: 4.13…

Thanks, very nice to see these results! Certainly using GPUs with more RAM makes things simpler to scale. Gradient accumulation is as easy as adding a counter for number of steps and an "if counter % gradient_accumulation_steps:` around `optimizer.step()`, so that can also be tried simply on a single GPU / cheaper GPUs. But if you can just use 8xA100 and your pipeline parallizes well, you also get results (almost) 8 times faster, which is certainly nicer to experiment of course!

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#109

Post by spi »

alansaber wrote: post #55
spi wrote: post #46

A separate comment about conclusions about why they are worse than OpenAI GPT2 - which to me feel to be missing the point. One main point is batch size - I'd agree with Gemini here. Batch size Training duration is definitely also a reason - models do get better over time, otherwise people wouldn't train so long wasting millions :-) just how long for optimality is unclear, but certainly The optimizer could also play a…

To caveat, smaller batch sizes are generally better for model stability, but we go bigger because it substantially speeds up training

Mmh not really. As OP shows, speed increases with larger batch size, but only initially, until the GPU has high enough utilization; then speed improvements flatten out (although you might get OOM before that and not "really" see the flat part). Using smaller batch size increases _noise_, so quite literally decreases stability. That might be good sometimes: in the limit case, if the batch is as large as your training set, you'll end up in local minima and not be able to get out of it. But this is true for toy datasets like MNIST, here it's an entirely different beast.

With such large corpora as the ones used here, and very noisy ones at that, gradient updates are very noisy and that can harm quality. Or anyway, common lore is that one needs pretty large batch size to have the language model improve steadily.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#110

Post by BubbleRings »

QuadmasterXLII wrote: post #103
BubbleRings wrote: post #35

> …reused its embedding matrix as the weights for the linear layer that projects the context vectors from the last Transformers layer into vocab space to get the logits. At first glance this claim sounds airtight, but it quietly collapses under its own techno-mythology. The so-called “reuse” of the embedding matrix assumes a fixed semantic congruence between representational space and output projection, an assumption…

The second paragraph is highly derivative of the adversarial turbo encabulator, which Schmithuber invented in the 90s. No citation of course.

Are you saying I should have attributed, or ChatGPT should have? I suppose I would have but my spurving bearings were rusty.
Post reply on HN