Now this is cool. and can be used for evil AI.
LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
51–60 of 124 posts
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#52A separate comment about conclusions about why they are worse than OpenAI GPT2 - which to me feel to be missing the point. One main point is batch size - I'd agree with Gemini here. Batch size Training duration is definitely also a reason - models do get better over time, otherwise people wouldn't train so long wasting millions :-) just how long for optimality is unclear, but certainly The optimizer could also play a…
OP here -- thanks! I'm in the process of doing some trains using the same code plus DDP on big Lambda Labs machines, and (within the bounds of what I can afford) will hopefully have some interesting results about all of those shortly.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#53Has anyone done something like this but with apple silicon instead of a graphics card? Training a small LLM on an M2-M5?
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#54> …reused its embedding matrix as the weights for the linear layer that projects the context vectors from the last Transformers layer into vocab space to get the logits. At first glance this claim sounds airtight, but it quietly collapses under its own techno-mythology. The so-called “reuse” of the embedding matrix assumes a fixed semantic congruence between representational space and output projection, an assumption…
I have no idea what you’ve just said, so here is my upvote.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#55A separate comment about conclusions about why they are worse than OpenAI GPT2 - which to me feel to be missing the point. One main point is batch size - I'd agree with Gemini here. Batch size Training duration is definitely also a reason - models do get better over time, otherwise people wouldn't train so long wasting millions :-) just how long for optimality is unclear, but certainly The optimizer could also play a…
To caveat, smaller batch sizes are generally better for model stability, but we go bigger because it substantially speeds up training
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#56I love the level of detail ( probably, because I see it less and less these days ). It genuinely makes me wonder if anyone tried training LLMs on their own writings ( assuming those bigger than 100+ pages ) and what the results were.
Fine-tuning on a small corpus can definitely get you good results with some care
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#57> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level sco…
For small models this is for sure the way forward, there are some great small datasets out there (check out the tiny stories dataset that limits vocab to a certain age but keeps core reasoning inherent in even simple language https://huggingface.co/datasets/roneneldan/TinyStories https://arxiv.org/abs/2305.07759 ) I have less concrete examples but my understanding is that dataset curation is for sure the way many imp…
OP here: one thing that surprised me in this experiment was that the model trained on the more curated FineWeb-Edu dataset was worse than the one trained on FineWeb. That is very counterintuitive to me.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#58> …reused its embedding matrix as the weights for the linear layer that projects the context vectors from the last Transformers layer into vocab space to get the logits. At first glance this claim sounds airtight, but it quietly collapses under its own techno-mythology. The so-called “reuse” of the embedding matrix assumes a fixed semantic congruence between representational space and output projection, an assumption…
Disclaimer: working and occasionally researching in the space.
The first paragraph is clear linear algebra terminology, the second looked like deeper subfield specific jargon and I was about to ask for a citation as the words definitely are real but the claim sounded hyperspecific and unfamiliar.
I figure a person needs 12 to 18 months of linear algebra, enough to work through Horn and Johnson's "Matrix Analysis" or the more bespoke volumes from Jeffrey Humpheries to get the math behind ML. Not necessarily to use AI/ML as a tech, which really can benefit from the grind towards commodification, but to be able to parse the technical side of about 90 to 95 percent of conference papers.
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#59> When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don't even know how this works at all. It’s [stuff] like stock tickers, symbols, it's a huge amount of slop and garbage from like all the corners of the internet Seems like there would be low hanging fruit in heavier pre processing then? Something deterministic like a reading level sco…
Data filtering. Dataset curation. Curriculum learning. All already in use. It's not sexy, it's not a breakthrough, but it does help.
> All already in use.
At the big labs that makes sense. Bit more puzzled by why it isn’t used in the toy projects. Certainly more complexity but seems like it would make a big difference
Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
#60Earlier quoted context omitted.
I understand the concern, but I also think there are benefits to this approach. And while I absolutely agree with you on the likeness part used for a company, at a personal level, I believe it could have a great impact ( and be of use ). And, more importantly, you can then control the disposition of your likeness appropriately ( via an old fashioned will ). As a society, we seem to have solutions for these situations…
Given the velocity of this industry and it being largely driven by corporations, how many individuals do you think will have control over their likeness vs their likeness being stored by some entity they did not explicitly consent towards? I appreciate your take, I just think it is not in line with the current trajectory outside of some unique HN posters and the like - and even they will probably wake up one day real…
You do have a point. That is why I am not pushing it as a general solution and frankly why I am not super keen on putting everything on github for everyone to see. If there is only one dark joke of the current times, it is that pressing agree somehow constitutes agreeing to legally consenting all sorts of invasive practices.
I would absolutely not suggest doing what I am doing to an average user.
edit: Frankly, just by thinking I am above average I might be inviting a more risky behavior.