Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fork of Facebook’s LLaMa model to run on CPU

github.com

Post reply on HN
Print view View original

41–50 of 178 posts

Re: Fork of Facebook’s LLaMa model to run on CPU

#41

Post by saurik »

aqme28 wrote: post #21
dsign wrote: post #15

The thing I like the most about the current AI wave is the pressure is putting on computing hardware. Yes, mobile phones with long battery lives are cool and all of that, but most cool things I like are locked behind huge computational requirements.

Crazy to me that as soon as one GPU wave is dying (crypto), another one is picking up slack.

One day we'll find out that all of the VR, crypto, and maybe now AI bubbles were nothing but conspiracies being driven by big-GPU to keep their share price up.

Re: Fork of Facebook’s LLaMa model to run on CPU

#42

Post by markasoftware »

ComplexSystems wrote: post #34
markasoftware wrote: post #14

Earlier quoted context omitted.

that's literally all I did (plus switching the tensor type). I'd imagine people are posting and upvoting this not because it's actually interesting code but rather just because it runs unexpectedly fast on consumer CPUs and it's not something they considered feasible before.

How are you getting this to run fast? I'm on a top of the line M1 MBP and getting 1 token every 8 minutes.

probably pytorch is very optimized to x86. It's likely using lots of SIMD and whatnot. I'm sure it's possible to get similar performance on m1 macs, but not with the current version of pytorch.

Do you have enough ram? (not swapping to disk)?

Re: Fork of Facebook’s LLaMa model to run on CPU

#43

Post by input_sh »

KyeRussell wrote: post #24
mrtksn wrote: post #9

Unlike Stable Diffusion, I don't stumble upon people who actually use it. Are there examples of the output this can generate? What happens once you manage to run the model?

Pretty sure you wouldn’t see anyone using it commercially as IIRC it’s only public due to a leak.

It's not a leak, it's a shortcut.

You can download it from Facebook, but it's behind "apply for access" form. Magnet links floating around are just a workaround around that form.

That said, commercial use is forbidden by the license specified in the form: https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z...

Re: Fork of Facebook’s LLaMa model to run on CPU

#44

Post by favaq »

aqme28 wrote: post #21
dsign wrote: post #15

The thing I like the most about the current AI wave is the pressure is putting on computing hardware. Yes, mobile phones with long battery lives are cool and all of that, but most cool things I like are locked behind huge computational requirements.

Crazy to me that as soon as one GPU wave is dying (crypto), another one is picking up slack.

Gee, it's almost as if GPUs are useful.

Re: Fork of Facebook’s LLaMa model to run on CPU

#45

Post by asplake »

raverbashing wrote: post #27
markasoftware wrote: post #11

Earlier quoted context omitted.

I tested on a decidedly above average CPU, and got several words per second on the 7B model. I'd guess maybe one word per second on a more average one?

Cool so we're back to the days of 2400 baud modems

Simulating a slow typist

Re: Fork of Facebook’s LLaMa model to run on CPU

#46

Post by mylidlpony »

mrtksn wrote: post #39
notpushkin wrote: post #33

Earlier quoted context omitted.

4) Lukashenka grabs your profits and kicks you out ;-)

Why would he do that? More likely scenario would be "gets his share as a an apartment in London".

That's the oldest authoritarian trick in the book, pretty much any successful business in russia got the same fate for example. They even tried it with nginx.

Re: Fork of Facebook’s LLaMa model to run on CPU

#48

Post by toxicFork »

saurik wrote: post #41
aqme28 wrote: post #21

Earlier quoted context omitted.

Crazy to me that as soon as one GPU wave is dying (crypto), another one is picking up slack.

One day we'll find out that all of the VR, crypto, and maybe now AI bubbles were nothing but conspiracies being driven by big-GPU to keep their share price up.

The universe was a hoax invented by a GPU company

Re: Fork of Facebook’s LLaMa model to run on CPU

#49

Post by qingdao99 »

mrtksn wrote: post #28
KyeRussell wrote: post #24

Earlier quoted context omitted.

Pretty sure you wouldn’t see anyone using it commercially as IIRC it’s only public due to a leak.

I wasn't looking for a commercial use but its an Interesting point. Would it be possible to prove that someone is using it commercially? 1) Spin it up on a cluster in Belarus 2) ??? 3) Profit?

It could be watermarked
Post reply on HN