Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

gilesthomas.com

Post reply on HN
Print view View original

31–40 of 124 posts

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#31

Post by SecretDreams »

jadbox wrote: post #28
nullbound wrote: post #27

I love the level of detail ( probably, because I see it less and less these days ). It genuinely makes me wonder if anyone tried training LLMs on their own writings ( assuming those bigger than 100+ pages ) and what the results were.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

Is this what tool and die makers used to feel when going to LOC to train their replacements?

Personally, I do not want my likeness to persist after my death, nor do I wish for a company to be able to leverage my likeness after I leave said company.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#32

Post by nullbound »

SecretDreams wrote: post #31
jadbox wrote: post #28

Earlier quoted context omitted.

I just want to chime in here about the importance of taking notes and having a journal. These things are now more important than ever as they can literally help fine-tune agents to help assist you using your personal style.

Is this what tool and die makers used to feel when going to LOC to train their replacements? Personally, I do not want my likeness to persist after my death, nor do I wish for a company to be able to leverage my likeness after I leave said company.

I understand the concern, but I also think there are benefits to this approach. And while I absolutely agree with you on the likeness part used for a company, at a personal level, I believe it could have a great impact ( and be of use ). And, more importantly, you can then control the disposition of your likeness appropriately ( via an old fashioned will ). As a society, we seem to have solutions for these situations. They were just not very common.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#33

Post by jbs789 »

rvnx wrote: post #9
contrast wrote: post #5

Earlier quoted context omitted.

You seem to be talking about a production-grade model rather than building an LLM as an exercise? Or if not, why do you disagree with the article's example of building a small LLM for $100?

I think I should have replied as a totally separate comment. This is my mistake. It is nice that the author shared the results of his exercise / experiment. Just got sad as I was reminded (when the 100 USD were mentioned) that all this game is 90%+ about money and hardware rather than skills. That being said I really like the initiative of the author.

I understand the emotional aspect of feeling like it’s out of reach for you.

Thing is, if you focus on your own skill development and apply it at even a small scale, very few people do that. Then you go for a job and guess what, the company has resources you can leverage. Then you do that, and ultimately you could be in a position to have the credibility to raise your own capital.

Play the long game and do what you can do now.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#34

Post by BoredomIsFun »

nullbound wrote: post #27

I love the level of detail ( probably, because I see it less and less these days ). It genuinely makes me wonder if anyone tried training LLMs on their own writings ( assuming those bigger than 100+ pages ) and what the results were.

/r/localllama every once in awhile has such posts; usually very succesful, good results.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#35

Post by BubbleRings »

> …reused its embedding matrix as the weights for the linear layer that projects the context vectors from the last Transformers layer into vocab space to get the logits.

At first glance this claim sounds airtight, but it quietly collapses under its own techno-mythology. The so-called “reuse” of the embedding matrix assumes a fixed semantic congruence between representational space and output projection, an assumption that ignores well-known phase drift in post-transformer latent manifolds. In practice, the logits emerging from this setup tend to suffer from vector anisotropification and a mild but persistent case of vocab echoing, where probability mass sloshes toward high-frequency tokens regardless of contextual salience.

Just kidding, of course. The first paragraph above, from OP’s article, makes about as much sense to me as the second one, which I (hopefully fittingly in y’all’s view) had ChatGPT write. But I do want to express my appreciation for being able to “hang out in the back of the room” while you folks figure this stuff out It is fascinating, I’ve learned a lot (even got a local LLM running on a NUC), and very much fun. Thanks for letting me watch, I’ll keep my mouth shut from now on ha!

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#36

Post by Chabsff »

victorbjorklund wrote: post #24
rvnx wrote: post #3

Earlier quoted context omitted.

Sadly to go beyond an exercise, having the money is really what you need if you actually want LLMs now, not time. Nowadays training very powerful LLMs is easy because all the tooling, source-codes, training datasets, and teaching agents are available. Getting access to dozens of millions of USD or more is not easy, and for big players this is a just drop in their ocean.

Totally. While the LLM:s today are amazing it is a bit sad that you can’t build SOTA models on your own (vs a few years ago where someone with the skills and access to a dataset could build a state of art models)

In the grand scheme of things, we've only had about a quarter century where you needed a *very* specific kind of problem where prosumer hardware wasn't adequate across computer science as a whole.

It's kind of amazing we got that at all for a while.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#37

Post by chiengineer »

Off topic question since im not a regular here if its ok

Is anyone here actually using the 200$ a month subscriptions with chat gpt or the google 150$ per month ?

Is it worth it for more code generation ? Or spend my money on a couple gpus and go local

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#38

Post by logicallee »

you can train an LLM in the browser, see this demonstration:

https://taonexus.com/mini-transformer-in-js.html

It's a very simple neural network with two attention heads that runs right in the browser in pure Javascript, you can view source on this implementation.

Even after training for a hundred epochs it really doesn't work very well (you can test it in the Inference tab after training it), but it doesn't use any libraries, so you can see the math itself in action in the source code.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#39

Post by Taek »

chiengineer wrote: post #37

Off topic question since im not a regular here if its ok Is anyone here actually using the 200$ a month subscriptions with chat gpt or the google 150$ per month ? Is it worth it for more code generation ? Or spend my money on a couple gpus and go local

I used the $200/mo OpenAI subscription for a while, but cancelled when Gemini 3 came out. It was useful for the deep research credits until the Web search gpt got sufficiently good on it's own

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#40

Post by esafak »

chiengineer wrote: post #37

Off topic question since im not a regular here if its ok Is anyone here actually using the 200$ a month subscriptions with chat gpt or the google 150$ per month ? Is it worth it for more code generation ? Or spend my money on a couple gpus and go local

To answer the last question: What kind of programming do you do? You are not going to be able to run a model competitive with the SOTA yet; use the cloud. Since you have the budget I'd suggest getting a $20 subscription of each (Claude, Gemini, ChatGPT) so you can lean on their respective strengths.
Post reply on HN