Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fork of Facebook’s LLaMa model to run on CPU

github.com

Post reply on HN
Print view View original

121–130 of 178 posts

Re: Fork of Facebook’s LLaMa model to run on CPU

#121

Post by counttheforks »

varispeed wrote: post #109
atleastoptimal wrote: post #25

Earlier quoted context omitted.

Which is a good thing. So glad all that GPU compute is being used on cool stuff rather than running SHA-256 18 quintillion times

> running SHA-256 18 quintillion times or games. People could have been studying or doing something more important than wasting time and energy. I get that it is entertainment, but so are board games and that don't require mining rare earth minerals or putting pressure on the grid as you can always play board games with candles on.

Or TV shows or movies. People could have been studying or doing something more important than wasting time and energy.

Or going outside. People could have been studying or doing something more important than wasting time and energy.

Or not being locked in the education facility. People could have been studying or doing something more important than wasting time and energy.

Re: Fork of Facebook’s LLaMa model to run on CPU

#122

Post by UncleEntity »

Xerox9213 wrote: post #115
speedgoose wrote: post #100

Earlier quoted context omitted.

Can you cite an useful result? I can't but I don't think that some people getting richer is useful.

https://www.elliptic.co/blog/live-updates-millions-in-crypto... https://www.cnbc.com/2022/03/23/ukrainian-flees-to-poland-wi... In general, the purpose of Bitcoin is not to get rich, but to have a currency that is universally accepted and not tied to a political party’s fiscal decisions.

In general, so is the Federal Reserve.

Re: Fork of Facebook’s LLaMa model to run on CPU

#123

Post by sva_ »

cinntaile wrote: post #8
Zetobal wrote: post #3

Earlier quoted context omitted.

I guess the simple fact that it didn't before his patch?

Usually you just trivially have the model run on cpu or gpu by simply writing .cpu() at specific places, so he's wondering why this isn't the case here.

Or better yet, define a device = 'cpu', and use tensor.to(device).

Re: Fork of Facebook’s LLaMa model to run on CPU

#124

Post by gpm »

haolez wrote: post #120

Would it be possible to run the 65B one like this as well? Is the bottleneck just the RAM, or would I need an absurd number of CPUs as well? It's not that hard to create a consumer-grade desktop with 256GB in 2023.

I don't know about this fork specifically, but in general yes absolutely.

Even without enough ram, you can stream model weights from disk and run at [size of model/disk read speed] seconds per token.

I'm doing that on a small GPU with this code, but it should be easy to get this working with the CPU as compute instead (and at least with my disk/CPU, I'm not even sure that it would run even slower, I think disk read would probably still be the bottleneck)

A lack of an absurd number of CPUs just means it's slow, not impossible.

https://github.com/gmorenz/llama/tree/ssd

Re: Fork of Facebook’s LLaMa model to run on CPU

#125

Post by downvotetruth »

haolez wrote: post #120

Would it be possible to run the 65B one like this as well? Is the bottleneck just the RAM, or would I need an absurd number of CPUs as well? It's not that hard to create a consumer-grade desktop with 256GB in 2023.

You don't need 256 GB. A pair of the new 48GB DDR5 will work along with a pair of 32GB sticks should work in a consumer DDR5 MB to fit the weights. It does burst when initially loading. So, a fast disk with about the same swap size as RAM seems necessary. It took about 25 mins to generate a single 500 character response using a 5800X & 32 GB DDR4, but I was not able to get to it to run on more than 1 thread with the 7B model.

Re: Fork of Facebook’s LLaMa model to run on CPU

#126

Post by haolez »

gpm wrote: post #124
haolez wrote: post #120

Would it be possible to run the 65B one like this as well? Is the bottleneck just the RAM, or would I need an absurd number of CPUs as well? It's not that hard to create a consumer-grade desktop with 256GB in 2023.

I don't know about this fork specifically, but in general yes absolutely. Even without enough ram, you can stream model weights from disk and run at [size of model/disk read speed] seconds per token. I'm doing that on a small GPU with this code, but it should be easy to get this working with the CPU as compute instead (and at least with my disk/CPU, I'm not even sure that it would run even slower, I think disk read w…

Yeah, I find this area fascinating. Like, it's very cool to run a 7B params model locally, but it must feel like a toy when compared to ChatGPT, for example.

However, the 65B parameter, according to the benchmarks, is such a beast that you might be able to do some things on it that are not possible on ChatGPT (despite all of ChatGPT's quality of life features). Amazing times.

Re: Fork of Facebook’s LLaMa model to run on CPU

#127

Post by haolez »

downvotetruth wrote: post #125
haolez wrote: post #120

Would it be possible to run the 65B one like this as well? Is the bottleneck just the RAM, or would I need an absurd number of CPUs as well? It's not that hard to create a consumer-grade desktop with 256GB in 2023.

You don't need 256 GB. A pair of the new 48GB DDR5 will work along with a pair of 32GB sticks should work in a consumer DDR5 MB to fit the weights. It does burst when initially loading. So, a fast disk with about the same swap size as RAM seems necessary. It took about 25 mins to generate a single 500 character response using a 5800X & 32 GB DDR4, but I was not able to get to it to run on more than 1 thread with the…

Why? Is it a limitation of the model or just something with the configuration that you couldn't figure out for this test?

Re: Fork of Facebook’s LLaMa model to run on CPU

#128

Post by basch »

haolez wrote: post #120

Would it be possible to run the 65B one like this as well? Is the bottleneck just the RAM, or would I need an absurd number of CPUs as well? It's not that hard to create a consumer-grade desktop with 256GB in 2023.

I wonder if we will start to see complex prune functions and tools start to pop up.

So before you start a task, you sort of describe the domain, and the model is separated into the third most useful and relevant to that topic/query, and 2/3rd most distant from that realm. Then either just the 1/3rd is used in a detached fashion, or it works as 2 layers of cache, one in ram one on disk.

Re: Fork of Facebook’s LLaMa model to run on CPU

#129

Post by EVa5I7bHFq9mnYK »

schaefer wrote: post #71
saurik wrote: post #41

Earlier quoted context omitted.

One day we'll find out that all of the VR, crypto, and maybe now AI bubbles were nothing but conspiracies being driven by big-GPU to keep their share price up.

Speaking for myself, I have already gotten more use out of 2 weeks of chatgpt than I have out of 16 years of Bitcoin

14. First Bitcoin was mined 14 years ago. And Bitcoins have not been mined with GPUs since 2013.

Re: Fork of Facebook’s LLaMa model to run on CPU

#130

Post by mrtranscendence »

varispeed wrote: post #109
atleastoptimal wrote: post #25

Earlier quoted context omitted.

Which is a good thing. So glad all that GPU compute is being used on cool stuff rather than running SHA-256 18 quintillion times

> running SHA-256 18 quintillion times or games. People could have been studying or doing something more important than wasting time and energy. I get that it is entertainment, but so are board games and that don't require mining rare earth minerals or putting pressure on the grid as you can always play board games with candles on.

Monopoly by candlelight, just the future I had always envisioned.
Post reply on HN