Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history of LLaMA models

agi-sphere.com

Post reply on HN
Print view View original

61–70 of 87 posts

Re: A brief history of LLaMA models

#61

Post by microtonal »

rch wrote: post #12
jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

Just use Nixpkgs already.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly.

The only recourse is using the -bin flavors of PyTorch, etc. which will just download the precompiled upstream versions. Sadly, the result will still be much slower than other distributions. First because Python isn't compiled with optimizations and LTO in nixpkgs by default, because it is not reproducible. So, you override the Python derivation to enable optimizations and LTO. Python builds fine, but to get the machine learning ecosystem on you machine, Nix needs to build a gazillion Python packages, since the derivation hash of Python changed. Turns out that many derivations don't actually build. They build with the little amount of parallelism available on Hydra builders, but many Python packages will fail to build because of concurrency issues in tests that do manifest on your nice 16 core machine.

So, you spend hours fixing derivations so that they build on many core machines and upstream all the diffs. Or YOLO and you disable unit tests altogether. A few hours/days later (depending on your knowledge of Nix), you finally have a built of all packages that you want, you launch whatever you are doing on your CUDA-capable GPU. Turns out that it is 30-50% slower. Finding out why is another multi-day expedition in profiling and tinkering.

In the end pyenv (or a Docker container) on a boring distribution doesn't look so bad.

(Disclaimer: I initially added the PyTorch/libtorch bin packages to nixpkgs and was co-maintainer of the PyTorch derivation for a while.)

Re: A brief history of LLaMA models

#62

Post by doodlesdev »

esquire_900 wrote: post #47
hewlett wrote: post #32

Earlier quoted context omitted.

You can either spend $5 per month for VPS for a webserver for your static blog which you now have to secure properly, or you can just stick it on Cloudflare Pages for free

Cloudflare bot protection adds a slight delay (as in seconds) at best, and completely blocks users like the parent comment at worst. It costs no money, but isn't free either.

We are talking about different products though, I believe you can have your webpage hosted on Cloudflare Pages or behind Cloudflare CDN without enabling invasive "bot" detection.

Re: A brief history of LLaMA models

#63

Post by kolinko »

jiggawatts wrote: post #50
KETpXDDzR wrote: post #11

Earlier quoted context omitted.

llama.cpp was easy to setup IMO

Can you link to a working Dockerfile? I've heard several people say that it is easy, but then surely it ought to be trivial to set script the build so that it works reliable in a container!

but what for?

Re: A brief history of LLaMA models

#64

Post by kolinko »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

If you can use it for research/purpose purposes, you just do, and if you manage to build something good enough, you can get funding to train your own model then.

Also, with business there are few "can do / can't do" - it's about managing risks. If a penalty for doing is negligible (FB cannot catch you abusing license in private), from a business standpoint there is no issue in doing so - especially with things that are ethically kind-of-ok.

Re: A brief history of LLaMA models

#65

Post by yoquan »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

Because it's fun :-) And effort to bring up a could-be-commercial version is on going.

https://www.together.xyz/blog/redpajama

Re: A brief history of LLaMA models

#66

Post by UncleEntity »

celdon25 wrote: post #46
UncleEntity wrote: post #38

Earlier quoted context omitted.

They require someone to take time out of their busy schedule to build? Complaining that people won’t work for you for free is a bit much, don’t you think?

I remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community). Now it is apparently seen as "working for free for ungrateful people"

There’s a big difference between making a project easy to use and requiring they package it all up in someone’s preferred container format which can’t even legally include all the dependencies.

Facebook seems to be pretty hands off (as is expected since the code is open source) unless you distribute the model weights and then they drop the dmca banhammer.

So, yeah, simply complaining with no effort to understand the problem is kind of ungrateful.

Re: A brief history of LLaMA models

#67

Post by UncleEntity »

microtonal wrote: post #61
rch wrote: post #12

Earlier quoted context omitted.

Just use Nixpkgs already.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only re…

I learned the very hard way not to mess with the python version the system depends on.

If you absolutely must then build it separately and link (or use) that exactly like blender does with their binaries. Campbell (one of the core blender devs) used to love to bump the python version as soon as it was released and if you wanted to do any dev work you’d have to run another python environment until the distro version caught up. Being as I liked to use the fedora libs as a sort of sanity check this was a bit of a hassle to say the least.

Re: A brief history of LLaMA models

#68

Post by MakeUsersWant »

valine wrote: post #26
jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.

Could you publish a set of known-good versions (pip freeze, OS version, etc)?

Re: A brief history of LLaMA models

#69

Post by nl »

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

Use the HuggingFace Transformer library. Unlike random github repos they are professionally maintained with proper versioning.

Here's the docs: https://huggingface.co/docs/transformers/main/model_doc/llam...

Re: A brief history of LLaMA models

#70

Post by Labo333 »

vessenes wrote: post #13

Most places that recommend llama.cpp for mac fail to mention https://github.com/jankais3r/LLaMA_MPS , which runs unquantized 7b and 13b models on the M1/M2 GPU directly. It's slightly slower, (not a lot), and significantly lower energy usage. To me the win not having to quantize while not melting a hole in my lap is huge; I wish more people knew about it.

Can you explain why they have a "significantly lower energy usage"? Thanks!
Post reply on HN