Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history of LLaMA models

agi-sphere.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: A brief history of LLaMA models

#81

Post by alchemist1e9 »

microtonal wrote: post #79
alchemist1e9 wrote: post #77

Earlier quoted context omitted.

As a heavy nixpkgs user your comment resonates and makes me nervous. I was thinking if it is possible in nixpkgs to create a branch that attempts to create a version match to specific distributions, especially Ubuntu as the ML world is most using it. My idea is to somehow use the deb package information to “shadow” another distribution. > First because Python isn't compiled with optimizations and LTO in nixpkgs by de…

I understand your comments including above and the one about CUDA binaries. Just one clarification on the concurrency in tests failure, do you mean it overloads the machine running multi process tests that then tests fail due to assumptions by the package authors? I've found that quite some test suites have race conditions (e.g. simultaneous modification of files, etc.), which manifest themselves e.g. when a package…

Have you ever come across anything debian -> nix expression tools?

Ubuntu seems to be winning mindshare across the board and while this would be different than nixpkgs itself I was thinking if it is possible to mass convert deb packages into nix expressions, this combined with overlays would allow rapid incremental testing of marginal modifications to a current distribution’s stacking of versions.

A bit like how Nix community has tools on top of the various language packaging systems but this would be a layer on top of the debian packaging standards.

Maybe it’s crazy but just an idea I’ve been having recently and wondering how hard it might be. Importantly debian deb and apt systems are very reproducible and structured which is a good fit for a Nix based layer.

Re: A brief history of LLaMA models

#82

Post by dventimi »

alchemist1e9 wrote: post #76
jiggawatts wrote: post #55

Earlier quoted context omitted.

> specialize in ML research not software engineering. That has nothing to do with Python tooling being bad. A safe assumption is that Python package managers are being developed by developers , who have no excuse. If a C++ codebase developed by scientists had null pointer exceptions in it, then I could excuse things. But if the C++ compiler itself introduced unforced null pointer errors, then it absolutely deserves c…

One point you have I absolutely agree with: > Conda is at fault here Conda is so so bad. But trying to explain why to people who have fallen into it’s trap is difficult. People don’t realize the packages are not signed on enough information to reproduce them. The optimizer to find matching versions to make an environment that satisfies your constraints is really bad idea. As an experienced C++ developer unfortunately…

What's a good alternative? No, really. Conda has caused me so much pain.

Re: A brief history of LLaMA models

#83

Post by vessenes »

Labo333 wrote: post #70
vessenes wrote: post #13

Most places that recommend llama.cpp for mac fail to mention https://github.com/jankais3r/LLaMA_MPS , which runs unquantized 7b and 13b models on the M1/M2 GPU directly. It's slightly slower, (not a lot), and significantly lower energy usage. To me the win not having to quantize while not melting a hole in my lap is huge; I wish more people knew about it.

Can you explain why they have a "significantly lower energy usage"? Thanks!

Yes. Llama.cpp uses the CPU to do inference. MPS is the GPU for the macbook, so it has highly performant cores which can be used to do the computation. When you get inference done on the GPU, there's no (less?) energy wasted on general compute type work. :)

Re: A brief history of LLaMA models

#84

Post by noman-land »

detrites wrote: post #60
noman-land wrote: post #36

Earlier quoted context omitted.

I've been meaning to ask this question as an LLM noob but what exactly is quantizing in this context and why do people do it? I know of quantizing in the digital audio context only.

Models in this context are just a big list of numbers. The numbers will have a native "type", for example 32-bit floats. These are numbers like 0.7373663777 or -1.000003663. The 32-bit float type can represent something like 4.3 billion numbers (sort of). It was discovered though, that while models may need this level of precision when creating them ("training"), they don't need it nearly as much after the fact, when…

Thanks so much for this explanation. It makes perfect sense and is really simple.

Re: A brief history of LLaMA models

#85

Post by rch »

microtonal wrote: post #61
rch wrote: post #12

Earlier quoted context omitted.

Just use Nixpkgs already.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only re…

Granted, but I already have to run isolated container registries, pypi, maven, terraform, CI/CD, etc., etc., and so locally addressing the problems you've described is unavoidable and will realize significant efficiencies in any case. Everything about working in partially or fully air gapped environments is painful - no surprises there.

But I also think it's fine for individuals and researchers working in ML to expect some extra compiling, as long as the outcome is reliable. I'm stuck at home this weekend resurrecting an analysis from 10+ years ago, complete with Python, R, Java, and Fortran dependencies^, and I'm definitely wishing I'd known about Nix back then.

^btw, thanks to whomever included hdf5-mpi in Nixpkgs. Your work is greatly appreciated.

Re: A brief history of LLaMA models

#86

Post by alchemist1e9 »

dventimi wrote: post #82
alchemist1e9 wrote: post #76

Earlier quoted context omitted.

One point you have I absolutely agree with: > Conda is at fault here Conda is so so bad. But trying to explain why to people who have fallen into it’s trap is difficult. People don’t realize the packages are not signed on enough information to reproduce them. The optimizer to find matching versions to make an environment that satisfies your constraints is really bad idea. As an experienced C++ developer unfortunately…

What's a good alternative? No, really. Conda has caused me so much pain.

As I mentioned I’ve settled on Nix with nixpkgs however it’s got a steep learning curve and really isn’t appropriate for anyone but fairly experienced unix hackers.

It’s a problem.

Re: A brief history of LLaMA models

#87

Post by rch »

rch wrote: post #85
microtonal wrote: post #61

Earlier quoted context omitted.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only re…

Granted, but I already have to run isolated container registries, pypi, maven, terraform, CI/CD, etc., etc., and so locally addressing the problems you've described is unavoidable and will realize significant efficiencies in any case. Everything about working in partially or fully air gapped environments is painful - no surprises there. But I also think it's fine for individuals and researchers working in ML to expec…

I'll add that I do end up disabling tests more than I'd like, even for very popular packages. Point taken there.
Post reply on HN