A brief history of LLaMA models • Hacker News Board

Re: A brief history of LLaMA models

#11

Post by KETpXDDzR » Sat, Apr 29, 2023, 10:32 PM UTC

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

llama.cpp was easy to setup IMO

Re: A brief history of LLaMA models

#12

Post by rch » Sat, Apr 29, 2023, 10:34 PM UTC

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

Just use Nixpkgs already.

Re: A brief history of LLaMA models

#13

Post by vessenes » Sat, Apr 29, 2023, 10:35 PM UTC

Most places that recommend llama.cpp for mac fail to mention https://github.com/jankais3r/LLaMA_MPS, which runs unquantized 7b and 13b models on the M1/M2 GPU directly. It's slightly slower, (not a lot), and significantly lower energy usage. To me the win not having to quantize while not melting a hole in my lap is huge; I wish more people knew about it.

Re: A brief history of LLaMA models

#14

Post by [deleted] » Sat, Apr 29, 2023, 10:43 PM UTC

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

[deleted]

Re: A brief history of LLaMA models

#15

Post by Taek » Sat, Apr 29, 2023, 10:45 PM UTC

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

I have it running locally using the oobabooga webui, setup was moderately annoying but I'm definitely no python expert and I didn't have too much trouble.

Re: A brief history of LLaMA models

#16

Post by doodlesdev » Sat, Apr 29, 2023, 10:48 PM UTC

   > Our system thinks you might be a robot!
   We're really sorry about this, but it's getting harder and harder to tell the difference between humans and bots these days.

Yeah, fuck you too. Come on, really, why put this in front of a _blog post_? Is it that hard to keep up with the bot requests when serving a static page?

Re: A brief history of LLaMA models

#17

Post by int_19h » Sat, Apr 29, 2023, 10:51 PM UTC

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

All of these things exist in the Python package ecosystem, and are generally much more common outside of ML/DS stuff. The latter... well, it reminds me of coding in early PHP days. Basically, anything goes so long as it works.

Re: A brief history of LLaMA models

#18

Post by sagarm » Sat, Apr 29, 2023, 10:53 PM UTC

sp332 wrote: post #7

newswasboring wrote: post #5

Earlier quoted context omitted.

With all these new AI models, both stable diffusion and llama specially, I'm considering switching to iPhone. I don't think I fully understand why iPhones and Macs are getting so many implementations but it seems like it's hardware based.

iPhones leaned in to "computational photography" a long time ago. Eventually they added custom hardware to handle all the matrix multiplies efficiently. They exposed some of it to apps with an API called CoreML. They've been adding more features like on-device photo tagging, voice recognition, VR stuff.

Google was the leader on computational smartphone photography. They released their "night sight" mode before Samsung and Apple had anything competitive.

Re: A brief history of LLaMA models

#19

Post by int_19h » Sat, Apr 29, 2023, 10:55 PM UTC

simonw wrote: post #8

newswasboring wrote: post #5

Earlier quoted context omitted.

With all these new AI models, both stable diffusion and llama specially, I'm considering switching to iPhone. I don't think I fully understand why iPhones and Macs are getting so many implementations but it seems like it's hardware based.

My understanding is that part of it is that Apple Silicon shares all available RAM between CPU and GPU. I'm not sure how many of these models are actively taking advantage of that architecture yet though.

The GPU isn't actually used by llama.cpp. What makes it that much faster is that the workload, either on CPU or on GPU, is very memory-intensive, so it benefits greatly from fast RAM. And Apple is using DDR5 running at very high clock speeds for this shared memory stuff.

It's still noticeably slower than GPU, though.

Re: A brief history of LLaMA models

#20

Post by mdaniel » Sat, Apr 29, 2023, 11:42 PM UTC

FloatArtifact wrote: post #4

There needs to be a slight dedicated to tracking all these models with regular updates.

Heh, there is, and you're on it. But a slightly more serious answer is that would be a good feature for Huggingface to add since they're the GitHub of models. I actually suggested to GitHub that they should allow contributions of the repo topics since a lot of developers don't know or don't bother to add topics to their repos, making discoverability harder than necessary. GH ignored it but maybe Huggingface could implement such a thing