Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history of LLaMA models

agi-sphere.com

Post reply on HN
Print view View original

51–60 of 87 posts

Re: A brief history of LLaMA models

#51

Post by forgingahead »

jiggawatts wrote: post #49
dagenix wrote: post #45

Earlier quoted context omitted.

There is a lot that can be improved with python packaging, but calling it "childish" is itself a pretty immature comment.

Is it? Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence . E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68... Take this line: pip3 install torch torchvision torchaudio Which version of torch is this? The latest. FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04 Which version of CUDA is this? An incompatible one…

You're right, Python's ecosystem and dependency management is a shitshow and everybody involved should be ashamed of themselves. But of course there are many "you're holding it wrong" commenters here who are in denial of this fact. It's an absolute pity that Python has become the de-facto language for public ML projects.

Re: A brief history of LLaMA models

#52

Post by dagenix »

jiggawatts wrote: post #49
dagenix wrote: post #45

Earlier quoted context omitted.

There is a lot that can be improved with python packaging, but calling it "childish" is itself a pretty immature comment.

Is it? Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence . E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68... Take this line: pip3 install torch torchvision torchaudio Which version of torch is this? The latest. FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04 Which version of CUDA is this? An incompatible one…

> you absolutely deserve to be labelled immature and even childish

Do you appreciate that people aren't making technical mistakes on purpose just to spite you? Or that maybe some of the folks writing these libraries are experts in fields other than dependency management? Are you an expert in all things? Would you find it helpful if someone identifies one thing that you aren't great at and then calls you names on the internet over that one thing?

There is a pretty significant difference between making a technical critique and just being rude. And being right about the former doesn't make the latter ok.

Re: A brief history of LLaMA models

#53

Post by dagenix »

forgingahead wrote: post #51
jiggawatts wrote: post #49

Earlier quoted context omitted.

Is it? Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence . E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68... Take this line: pip3 install torch torchvision torchaudio Which version of torch is this? The latest. FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04 Which version of CUDA is this? An incompatible one…

You're right, Python's ecosystem and dependency management is a shitshow and everybody involved should be ashamed of themselves. But of course there are many "you're holding it wrong" commenters here who are in denial of this fact. It's an absolute pity that Python has become the de-facto language for public ML projects.

> everybody involved should be ashamed of themselves

Have you ever read about how open source project leaders often experience a lot of toxicity and anxiety about trying to keep up with the users they are supporting for free? If not, I suggest you do since this is the exact type of comment that is hurtful and unhelpful.

Re: A brief history of LLaMA models

#54

Post by PostOnce »

jiggawatts wrote: post #50
KETpXDDzR wrote: post #11

Earlier quoted context omitted.

llama.cpp was easy to setup IMO

Can you link to a working Dockerfile? I've heard several people say that it is easy, but then surely it ought to be trivial to set script the build so that it works reliable in a container!

No need to drag a gigabyte of docker stuff into this, just extract the zip file from github and type make into your terminal

congratulations, it now works.

If you're not a developer, maybe you'll have to type sudo apt install build-essential first. Congratulations, now you too, a non-developer, are running it locally.

https://github.com/ggerganov/llama.cpp

Re: A brief history of LLaMA models

#55

Post by jiggawatts »

valine wrote: post #48
celdon25 wrote: post #46

Earlier quoted context omitted.

I remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community). Now it is apparently seen as "working for free for ungrateful people"

Most of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?

> specialize in ML research not software engineering.

That has nothing to do with Python tooling being bad. A safe assumption is that Python package managers are being developed by developers, who have no excuse.

If a C++ codebase developed by scientists had null pointer exceptions in it, then I could excuse things. But if the C++ compiler itself introduced unforced null pointer errors, then it absolutely deserves criticism.

It shouldn't be possible for a ML researcher to use Conda or whatever package manager in a way that despite using a formally specified "requirements.txt", it won't build a week later because of how loose the specification of module versions is allowed to be.

The Python attitude and more specifically Conda is at fault here, not the ML researching trying to get his job done.

Re: A brief history of LLaMA models

#56

Post by detrites »

dagenix wrote: post #53
forgingahead wrote: post #51

Earlier quoted context omitted.

You're right, Python's ecosystem and dependency management is a shitshow and everybody involved should be ashamed of themselves. But of course there are many "you're holding it wrong" commenters here who are in denial of this fact. It's an absolute pity that Python has become the de-facto language for public ML projects.

> everybody involved should be ashamed of themselves Have you ever read about how open source project leaders often experience a lot of toxicity and anxiety about trying to keep up with the users they are supporting for free? If not, I suggest you do since this is the exact type of comment that is hurtful and unhelpful.

You both seem right. While the base criticism is correct, the emotion attached to it is only an impediment. Python+ML seems apparently in a "move fast and break things" phase. We should expect rough edges and be calm if we're to address that well.

Re: A brief history of LLaMA models

#57

Post by Blahah »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

Because none of the interesting things I want to do with LLMs have anything to do with commercial uses. And because I don't care what a license says when I'm doing stuff in private. I just case whether I am practically able to do it.

Re: A brief history of LLaMA models

#58

Post by dagenix »

detrites wrote: post #56
dagenix wrote: post #53

Earlier quoted context omitted.

> everybody involved should be ashamed of themselves Have you ever read about how open source project leaders often experience a lot of toxicity and anxiety about trying to keep up with the users they are supporting for free? If not, I suggest you do since this is the exact type of comment that is hurtful and unhelpful.

You both seem right. While the base criticism is correct, the emotion attached to it is only an impediment. Python+ML seems apparently in a "move fast and break things" phase. We should expect rough edges and be calm if we're to address that well.

I agree with the person I'm responding to 100% on the technical merits.

Re: A brief history of LLaMA models

#59

Post by emikulic »

kmod wrote: post #33
jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

I believe the cuda stuff, via Nvidia licensing restrictions, is forced to live outside of these packaging systems (so that you sign a Nvidia eula). Not saying this is a good thing but I think that none of the systems you mentioned would handle this well either

This used to be the case but nowadays you can just pip install e.g. https://libraries.io/pypi/nvidia-cuda-runtime-cu11

Re: A brief history of LLaMA models

#60

Post by detrites »

noman-land wrote: post #36
vessenes wrote: post #13

Most places that recommend llama.cpp for mac fail to mention https://github.com/jankais3r/LLaMA_MPS , which runs unquantized 7b and 13b models on the M1/M2 GPU directly. It's slightly slower, (not a lot), and significantly lower energy usage. To me the win not having to quantize while not melting a hole in my lap is huge; I wish more people knew about it.

I've been meaning to ask this question as an LLM noob but what exactly is quantizing in this context and why do people do it? I know of quantizing in the digital audio context only.

Models in this context are just a big list of numbers. The numbers will have a native "type", for example 32-bit floats. These are numbers like 0.7373663777 or -1.000003663. The 32-bit float type can represent something like 4.3 billion numbers (sort of).

It was discovered though, that while models may need this level of precision when creating them ("training"), they don't need it nearly as much after the fact, when simply running them to get results ("inference").

So quantisation is the process of getting that big set of, say, 32-bit floats, and "mapping" them to a much smaller number type. Eg, an 8-bit integer ("INT8"). This is a number in the range 0-255 (or -128 to +127).

So, to quantise a list of 32-bit floats, you could go through the list and analyse. Maybe they're all in the range -1.0 to +1.0. Maybe there are many around the value of 0.99999 and 0.998 etc, so you decide to assign those the value "255" instead.

Repeat this until you've squashed that bunch of 32-bit values into 8-bits each. (Eg, maybe 0.750000 could be 192, etc.)

This could give a saving in memory footprint for the model of 4x smaller, and also makes it able to be run faster. So while you needed 16GB to run it before, now you might only need 4GB.

The expense is the model won't be as accurate. But, typically this is on the order of values like 90%, versus the memory savings of 4x. So it's deemed worth it.

It's through this process folks can run models that would normally require a 5-figure GPU to run, on their home machine, or even on the CPU, as it might be able to process integers easier and faster than floating point.

Post reply on HN