Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fork of Facebook’s LLaMa model to run on CPU

github.com

Post reply on HN
Print view View original

171–178 of 178 posts

Re: Fork of Facebook’s LLaMa model to run on CPU

#171

Post by ben_w »

TaylorAlexander wrote: post #166
ben_w wrote: post #96

Earlier quoted context omitted.

That's genuinely surprising. What sort of on-board compute do you typically have today?

The NVIDIA Jetson boards are popular, but even with a full desktop processor and state of the art GPU, you can easily down them in data from a LIDAR sensor or a few cameras. Especially since robots may also need fast response times. There is another reply to your comment that shares a lot of what I have experienced. You have so many pieces of code that need to run and a good handful of them are working on something l…

Thanks! :)

That definitely helps me understand why the footage of the robots in this video had to be sped up: https://youtu.be/Ybk8hxKeMYQ

Re: Fork of Facebook’s LLaMa model to run on CPU

#172

Post by Tepix »

downvotetruth wrote: post #125
haolez wrote: post #120

Would it be possible to run the 65B one like this as well? Is the bottleneck just the RAM, or would I need an absurd number of CPUs as well? It's not that hard to create a consumer-grade desktop with 256GB in 2023.

You don't need 256 GB. A pair of the new 48GB DDR5 will work along with a pair of 32GB sticks should work in a consumer DDR5 MB to fit the weights. It does burst when initially loading. So, a fast disk with about the same swap size as RAM seems necessary. It took about 25 mins to generate a single 500 character response using a 5800X & 32 GB DDR4, but I was not able to get to it to run on more than 1 thread with the…

All current Ryzen CPUs do not work with 48GB DDR5, right? That means if you want to go beyond 128GB you can get an old X399 board (there are some reports of people getting 256GB to work) or more recent Threadripper boards.

Re: Fork of Facebook’s LLaMa model to run on CPU

#174

Post by Tostino »

slaw wrote: post #168
idiotsecant wrote: post #165

Earlier quoted context omitted.

Watts per hour implies watts/hour. Watt-hour implies a number of watts multiplied by a length of time. Also known as energy. Watts are power. Watt hours are energy. Two different things. Watts/hour is nothing.

Watt/hour is speed of power. It doesn't make any sense.

No, you misunderstand what it means in that case. Watt-hours are comparable to joules. 1 watt-hour = 3600 joules.

Re: Fork of Facebook’s LLaMa model to run on CPU

#175

Post by slaw »

Tostino wrote: post #174
slaw wrote: post #168

Earlier quoted context omitted.

Watt/hour is speed of power. It doesn't make any sense.

No, you misunderstand what it means in that case. Watt-hours are comparable to joules. 1 watt-hour = 3600 joules.

watt*hour=joule

watt-hour: you cannot subtract time from power, it doesn't make any sense.

watt/hour: delta of power per time, something very weird.

Re: Fork of Facebook’s LLaMa model to run on CPU

#176

Post by downvotetruth »

Tepix wrote: post #172
downvotetruth wrote: post #125

Earlier quoted context omitted.

You don't need 256 GB. A pair of the new 48GB DDR5 will work along with a pair of 32GB sticks should work in a consumer DDR5 MB to fit the weights. It does burst when initially loading. So, a fast disk with about the same swap size as RAM seems necessary. It took about 25 mins to generate a single 500 character response using a 5800X & 32 GB DDR4, but I was not able to get to it to run on more than 1 thread with the…

All current Ryzen CPUs do not work with 48GB DDR5, right? That means if you want to go beyond 128GB you can get an old X399 board (there are some reports of people getting 256GB to work) or more recent Threadripper boards.

Current Ryzen CPUs do not work with either 24GB or 48GB DDR5.

Re: Fork of Facebook’s LLaMa model to run on CPU

#177

Post by popol12 »

fguerraz wrote: post #147
popol12 wrote: post #102

0.35 words/s on my 11th gen i5 with 7B model (framework laptop) not so bad !

How long did you have to wait for it to load? On my machine it's been running for 15mins, I'm still waiting for a prompt...

You get the full answer after completion, so it’s normal if you don’t get an output immediately

I computed the speed by doing speed=number of words/ total run time

Re: Fork of Facebook’s LLaMa model to run on CPU

#178

Post by popol12 »

kristianp wrote: post #163
fguerraz wrote: post #147

Earlier quoted context omitted.

How long did you have to wait for it to load? On my machine it's been running for 15mins, I'm still waiting for a prompt...

How much RAM do you both have?

32gb of ram and 64gb on NVME swap
Post reply on HN