how long for one token to infer on an average cpu?
I tested on a decidedly above average CPU, and got several words per second on the 7B model. I'd guess maybe one word per second on a more average one?
Fork of Facebook’s LLaMa model to run on CPU
11–20 of 178 posts
Re: Fork of Facebook’s LLaMa model to run on CPU
#12how long for one token to infer on an average cpu?
[deleted]
Re: Fork of Facebook’s LLaMa model to run on CPU
#13Since this is pytorch it should run on cpu anyway. What am I missing?
Reading the patch: https://github.com/facebookresearch/llama/compare/main...mar... Looks like this is just tweaking some defaults and commenting out some code that enables cuda. It also switches to something called gloo, which I'm not familiar with. Seems like an alternate backend.
you don't actually need to switch to gloo, I just have no idea what I'm doing.
Re: Fork of Facebook’s LLaMa model to run on CPU
#14Earlier quoted context omitted.
I guess the simple fact that it didn't before his patch?
Usually you just trivially have the model run on cpu or gpu by simply writing .cpu() at specific places, so he's wondering why this isn't the case here.
that's literally all I did (plus switching the tensor type). I'd imagine people are posting and upvoting this not because it's actually interesting code but rather just because it runs unexpectedly fast on consumer CPUs and it's not something they considered feasible before.
Re: Fork of Facebook’s LLaMa model to run on CPU
#15The thing I like the most about the current AI wave is the pressure is putting on computing hardware. Yes, mobile phones with long battery lives are cool and all of that, but most cool things I like are locked behind huge computational requirements.
Re: Fork of Facebook’s LLaMa model to run on CPU
#16Since this is pytorch it should run on cpu anyway. What am I missing?
Reading the patch: https://github.com/facebookresearch/llama/compare/main...mar... Looks like this is just tweaking some defaults and commenting out some code that enables cuda. It also switches to something called gloo, which I'm not familiar with. Seems like an alternate backend.
Gloo is a communication protocol for distributed computation (think along the lines of MPI)
Re: Fork of Facebook’s LLaMa model to run on CPU
#17Wondering how difficult this would be to get running on a m1 max?
Re: Fork of Facebook’s LLaMa model to run on CPU
#18The thing I like the most about the current AI wave is the pressure is putting on computing hardware. Yes, mobile phones with long battery lives are cool and all of that, but most cool things I like are locked behind huge computational requirements.
Agree. I work in robotics and we never have enough compute. I want to see us get to the point where the most advanced robot ever has all the compute it needs onboard, and that means huge growth in compute density and efficiency are needed.
Re: Fork of Facebook’s LLaMa model to run on CPU
#19Earlier quoted context omitted.
Usually you just trivially have the model run on cpu or gpu by simply writing .cpu() at specific places, so he's wondering why this isn't the case here.
that's literally all I did (plus switching the tensor type). I'd imagine people are posting and upvoting this not because it's actually interesting code but rather just because it runs unexpectedly fast on consumer CPUs and it's not something they considered feasible before.
That is vastly underestimating how tricky it is to make novel pieces of software run. There is a huge fringe of people who know how to click things but not use the terminal and a large fringe of people who know how to run "./execute.bat" but not how to write syntactically correct Python.
But a lot of those people want to play with LLMs.
Re: Fork of Facebook’s LLaMa model to run on CPU
#20i have a friend who owns an macbook pro m1 max. what kind of performance can i get?