Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history of LLaMA models

agi-sphere.com

Post reply on HN
Print view View original

31–40 of 87 posts

Re: A brief history of LLaMA models

#31

Post by Spivak »

api wrote: post #22
doodlesdev wrote: post #16

> Our system thinks you might be a robot! We're really sorry about this, but it's getting harder and harder to tell the difference between humans and bots these days. Yeah, fuck you too. Come on, really, why put this in front of a _blog post_? Is it that hard to keep up with the bot requests when serving a static page?

A lot of people just stick cloudflare in front of anything because of cargo cultism. A $5/mo VPS can serve a blog to tens of thousands of people unless you are running something stupidly inefficient. If it’s a static blog make that hundreds of thousands. For millions you might need to splurge on the $10 or $20 per month VPS.

Or you use the free thing and never think about it?

Re: A brief history of LLaMA models

#32

Post by hewlett »

api wrote: post #22
doodlesdev wrote: post #16

> Our system thinks you might be a robot! We're really sorry about this, but it's getting harder and harder to tell the difference between humans and bots these days. Yeah, fuck you too. Come on, really, why put this in front of a _blog post_? Is it that hard to keep up with the bot requests when serving a static page?

A lot of people just stick cloudflare in front of anything because of cargo cultism. A $5/mo VPS can serve a blog to tens of thousands of people unless you are running something stupidly inefficient. If it’s a static blog make that hundreds of thousands. For millions you might need to splurge on the $10 or $20 per month VPS.

You can either spend $5 per month for VPS for a webserver for your static blog which you now have to secure properly, or you can just stick it on Cloudflare Pages for free

Re: A brief history of LLaMA models

#33

Post by kmod »

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

I believe the cuda stuff, via Nvidia licensing restrictions, is forced to live outside of these packaging systems (so that you sign a Nvidia eula). Not saying this is a good thing but I think that none of the systems you mentioned would handle this well either

Re: A brief history of LLaMA models

#34

Post by jiggawatts »

valine wrote: post #26
jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.

> the trick is to search the project’s github issue tracker.

What ever happened to the crazy notion of Dockerfiles that simply build successfully?

Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?

Re: A brief history of LLaMA models

#35

Post by jiggawatts »

Spivak wrote: post #30
jiggawatts wrote: post #25

Earlier quoted context omitted.

I love how the response to a complaint about unreproducible builds without any versions being specified is an install script that straight up clones the "current commit" of a Git repo instead of a specific working commit id or tag. Astonishing.

There's some truth to the "Arch" philosophy for some types of software it's actually more stable to just pull from master.

Not in this case. I’ve tried several repos and they all fail with various version mismatch issues.

Re: A brief history of LLaMA models

#36

Post by noman-land »

vessenes wrote: post #13

Most places that recommend llama.cpp for mac fail to mention https://github.com/jankais3r/LLaMA_MPS , which runs unquantized 7b and 13b models on the M1/M2 GPU directly. It's slightly slower, (not a lot), and significantly lower energy usage. To me the win not having to quantize while not melting a hole in my lap is huge; I wish more people knew about it.

I've been meaning to ask this question as an LLM noob but what exactly is quantizing in this context and why do people do it? I know of quantizing in the digital audio context only.

Re: A brief history of LLaMA models

#37

Post by dragonwriter »

noman-land wrote: post #36
vessenes wrote: post #13

Most places that recommend llama.cpp for mac fail to mention https://github.com/jankais3r/LLaMA_MPS , which runs unquantized 7b and 13b models on the M1/M2 GPU directly. It's slightly slower, (not a lot), and significantly lower energy usage. To me the win not having to quantize while not melting a hole in my lap is huge; I wish more people knew about it.

I've been meaning to ask this question as an LLM noob but what exactly is quantizing in this context and why do people do it? I know of quantizing in the digital audio context only.

Quantization is reducing the precision (and size) of values.

https://huggingface.co/docs/optimum/concept_guides/quantizat...

Re: A brief history of LLaMA models

#38

Post by UncleEntity »

jiggawatts wrote: post #34
valine wrote: post #26

Earlier quoted context omitted.

I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.

> the trick is to search the project’s github issue tracker. What ever happened to the crazy notion of Dockerfiles that simply build successfully? Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?

They require someone to take time out of their busy schedule to build?

Complaining that people won’t work for you for free is a bit much, don’t you think?

Re: A brief history of LLaMA models

#39

Post by UncleEntity »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

Why can’t you use it for personal use?

I doubt the Facebook Police are going to bust down your door at 3am.

…or are they? peeks through curtains

Re: A brief history of LLaMA models

#40

Post by DustinBrett »

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

I had it running before with Dalai (https://github.com/cocktailpeanut/dalai) but have since moved to using the browser based WebGPU method (https://mlc.ai/web-llm/) which uses Vicuna 7B and is quite good.
Post reply on HN