What’s the rough idea of how this is possible? I thought you need the parrelism of a gpu
Fork of Facebook’s LLaMa model to run on CPU
101–110 of 178 posts
Re: Fork of Facebook’s LLaMa model to run on CPU
#1020.35 words/s on my 11th gen i5 with 7B model (framework laptop)
not so bad !
Re: Fork of Facebook’s LLaMa model to run on CPU
#103What’s the rough idea of how this is possible? I thought you need the parrelism of a gpu
inference has less pressure of parallelism compared to training
Re: Fork of Facebook’s LLaMa model to run on CPU
#104Re: Fork of Facebook’s LLaMa model to run on CPU
#105Re: Fork of Facebook’s LLaMa model to run on CPU
#106Wondering how difficult this would be to get running on a m1 max?
[deleted]
Re: Fork of Facebook’s LLaMa model to run on CPU
#107Earlier quoted context omitted.
On /g/ there's always a very active AI chatbot general that focuses on these models.
What's /g/?
https://4chan.org/g/
(enter at your own risk? I think it's kinda safe but still 4chan)
Re: Fork of Facebook’s LLaMa model to run on CPU
#108What’s the rough idea of how this is possible? I thought you need the parrelism of a gpu
[deleted]
Re: Fork of Facebook’s LLaMa model to run on CPU
#109Earlier quoted context omitted.
Crazy to me that as soon as one GPU wave is dying (crypto), another one is picking up slack.
Which is a good thing. So glad all that GPU compute is being used on cool stuff rather than running SHA-256 18 quintillion times
> running SHA-256 18 quintillion times
or games. People could have been studying or doing something more important than wasting time and energy. I get that it is entertainment, but so are board games and that don't require mining rare earth minerals or putting pressure on the grid as you can always play board games with candles on.
Re: Fork of Facebook’s LLaMa model to run on CPU
#110Earlier quoted context omitted.
Conda is gonna work much, much, much better for these kinds of applications, as that's what it's mostly used for, i.e. scientific/numerical computing with C/C++ dependencies.
Conda is an abomination that will download 4Gig of unnecessary shite and carelessly dump it into your system, thereby ruining your existing configuration in the process. Use it in a container or a VM unless you enjoy re-installing your system from scratch. Or better still, don't use it al all and let it wither away: these kind of braindead projects need to be put down with extreme prejudice.
The download size is large but conda doesn't ruin any existing configuration unless you explicitly tell it to be your native python environment. Conda is set up as a self-contained set of independent environments. Why would your system care what's inside the Anaconda directory unless you explicitly add it to your PATH/bash?