Fork of Facebook’s LLaMa model to run on CPU
1–10 of 178 posts
Re: Fork of Facebook’s LLaMa model to run on CPU
#2Since this is pytorch it should run on cpu anyway. What am I missing?
Re: Fork of Facebook’s LLaMa model to run on CPU
#3Since this is pytorch it should run on cpu anyway. What am I missing?
I guess the simple fact that it didn't before his patch?
Re: Fork of Facebook’s LLaMa model to run on CPU
#4Since this is pytorch it should run on cpu anyway. What am I missing?
Reading the patch: https://github.com/facebookresearch/llama/compare/main...mar...
Looks like this is just tweaking some defaults and commenting out some code that enables cuda. It also switches to something called gloo, which I'm not familiar with. Seems like an alternate backend.
Re: Fork of Facebook’s LLaMa model to run on CPU
#5Since this is pytorch it should run on cpu anyway. What am I missing?
[deleted]
Re: Fork of Facebook’s LLaMa model to run on CPU
#6how long for one token to infer on an average cpu?
Re: Fork of Facebook’s LLaMa model to run on CPU
#7It's useless before the model gets instruction and preference tunings. Won't even follow a simple ask, it will just assume it is a list of questions and generate more, or continue with slightly related comments.
FB trained a LLaMA-I (instruction tuned) variant for sports, just to show they can, but I don't think it got released.
Re: Fork of Facebook’s LLaMa model to run on CPU
#8Re: Fork of Facebook’s LLaMa model to run on CPU
#9Unlike Stable Diffusion, I don't stumble upon people who actually use it. Are there examples of the output this can generate? What happens once you manage to run the model?
Re: Fork of Facebook’s LLaMa model to run on CPU
#10how long for one token to infer on an average cpu?
From the readme: On a Ryzen 7900X, the 7B model is able to infer several words per second, quite a lot better than you'd expect!