Viewing profile — rasbt

Thanks for sharing!

I disagree here: Setting up a large-scale pretraining run is super complex if you have to manage your distributed computing platform, but looking at how the training data looks lik…

Agreed, understanding how a method works and how it would be done helps with developing an intuition for its limitations -- what it can and what it can't do

thanks for mentioning, that makes me super happy to hear!

I wouldn't pretrain from scratch, but continued pretraining is pretty popular for adapating LLMs to recent and/or custom data. (Sometimes this is referred to 'finetuning', however,…

Thanks, glad that this is helpful!

Quoting from the readme, it embraces other executers, including torch.compile and also works with multiple GPUs: > Thunder is a source-to-source compiler for PyTorch. It makes PyTo…

Yes, it's somewhat similar to the 2B model as it uses the same vocabulary size.

Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.