Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history of LLaMA models

agi-sphere.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: A brief history of LLaMA models

#71

Post by actualwitch »

hewlett wrote: post #32
api wrote: post #22

Earlier quoted context omitted.

A lot of people just stick cloudflare in front of anything because of cargo cultism. A $5/mo VPS can serve a blog to tens of thousands of people unless you are running something stupidly inefficient. If it’s a static blog make that hundreds of thousands. For millions you might need to splurge on the $10 or $20 per month VPS.

You can either spend $5 per month for VPS for a webserver for your static blog which you now have to secure properly, or you can just stick it on Cloudflare Pages for free

You don't need to host in on cloudflare, just use github pages - it's cdn is plenty fast.

Re: A brief history of LLaMA models

#72

Post by vidarh »

PostOnce wrote: post #54
jiggawatts wrote: post #50

Earlier quoted context omitted.

Can you link to a working Dockerfile? I've heard several people say that it is easy, but then surely it ought to be trivial to set script the build so that it works reliable in a container!

No need to drag a gigabyte of docker stuff into this, just extract the zip file from github and type make into your terminal congratulations, it now works. If you're not a developer, maybe you'll have to type sudo apt install build-essential first. Congratulations, now you too, a non-developer, are running it locally. https://github.com/ggerganov/llama.cpp

Did that, got a compiler error within seconds within seconds. Looks like it might need a newer version of gcc than is in the distro on my laptop. Which is why people ask for Docker. If it really will work just with make with build-essential on a new enough distro image, a Dockerfile that documents that would be trivial, and does not at all stop people from just typing make if their setup is new enough.

Re: A brief history of LLaMA models

#73

Post by microtonal »

UncleEntity wrote: post #67
microtonal wrote: post #61

Earlier quoted context omitted.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only re…

I learned the very hard way not to mess with the python version the system depends on. If you absolutely must then build it separately and link (or use) that exactly like blender does with their binaries. Campbell (one of the core blender devs) used to love to bump the python version as soon as it was released and if you wanted to do any dev work you’d have to run another python environment until the distro version c…

I learned the very hard way not to mess with the python version the system depends on.

That's not much of an issue on Nix. You can just override Python for your particular (machine learning) package set. The rest of the system will continue to use Python unmodified.

Re: A brief history of LLaMA models

#74

Post by joshvm »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

There are efforts to provide an open source replica of the training dataset and independently trained models. So far the dataset has been recreated following the original paper (allowing for some vagueness that Meta researchers didn't specify):

https://github.com/togethercomputer/RedPajama-Data/

https://twitter.com/togethercompute/status/16479179892645191...

Re: A brief history of LLaMA models

#75

Post by qaq »

jiggawatts wrote: post #55
valine wrote: post #48

Earlier quoted context omitted.

Most of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?

> specialize in ML research not software engineering. That has nothing to do with Python tooling being bad. A safe assumption is that Python package managers are being developed by developers , who have no excuse. If a C++ codebase developed by scientists had null pointer exceptions in it, then I could excuse things. But if the C++ compiler itself introduced unforced null pointer errors, then it absolutely deserves c…

Poetry is fairly reasonable package manager

Re: A brief history of LLaMA models

#76

Post by alchemist1e9 »

jiggawatts wrote: post #55
valine wrote: post #48

Earlier quoted context omitted.

Most of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?

> specialize in ML research not software engineering. That has nothing to do with Python tooling being bad. A safe assumption is that Python package managers are being developed by developers , who have no excuse. If a C++ codebase developed by scientists had null pointer exceptions in it, then I could excuse things. But if the C++ compiler itself introduced unforced null pointer errors, then it absolutely deserves c…

One point you have I absolutely agree with:

> Conda is at fault here

Conda is so so bad. But trying to explain why to people who have fallen into it’s trap is difficult. People don’t realize the packages are not signed on enough information to reproduce them. The optimizer to find matching versions to make an environment that satisfies your constraints is really bad idea.

As an experienced C++ developer unfortunately and fortunately I’ve concluded the most “correct” solution is to use nixpkgs.

Re: A brief history of LLaMA models

#77

Post by alchemist1e9 »

microtonal wrote: post #61
rch wrote: post #12

Earlier quoted context omitted.

Just use Nixpkgs already.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only re…

As a heavy nixpkgs user your comment resonates and makes me nervous.

I was thinking if it is possible in nixpkgs to create a branch that attempts to create a version match to specific distributions, especially Ubuntu as the ML world is most using it. My idea is to somehow use the deb package information to “shadow” another distribution.

> First because Python isn't compiled with optimizations and LTO in nixpkgs by default, because it is not reproducible. So, you override the Python derivation to enable optimizations and LTO. Python builds fine, but to get the machine learning ecosystem on you machine, Nix needs to build a gazillion Python packages, since the derivation hash of Python changed. Turns out that many derivations don't actually build. They build with the little amount of parallelism available on Hydra builders, but many Python packages will fail to build because of concurrency issues in tests that do manifest on your nice 16 core machine.

I understand your comments including above and the one about CUDA binaries. Just one clarification on the concurrency in tests failure, do you mean it overloads the machine running multi process tests that then tests fail due to assumptions by the package authors?

My main point is that Nix as a system is so incredibly powerful that perhaps there is an ability to “shadow” boring distributions, especially debian based, in some automated way. The we would have the best of both, baseline stability from the distribution and extensibility of nix.

Re: A brief history of LLaMA models

#78

Post by hardwaresofton »

detrites wrote: post #60
noman-land wrote: post #36

Earlier quoted context omitted.

I've been meaning to ask this question as an LLM noob but what exactly is quantizing in this context and why do people do it? I know of quantizing in the digital audio context only.

Models in this context are just a big list of numbers. The numbers will have a native "type", for example 32-bit floats. These are numbers like 0.7373663777 or -1.000003663. The 32-bit float type can represent something like 4.3 billion numbers (sort of). It was discovered though, that while models may need this level of precision when creating them ("training"), they don't need it nearly as much after the fact, when…

This is an excellent explanation, thank you

Re: A brief history of LLaMA models

#79

Post by microtonal »

alchemist1e9 wrote: post #77
microtonal wrote: post #61

Earlier quoted context omitted.

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only re…

As a heavy nixpkgs user your comment resonates and makes me nervous. I was thinking if it is possible in nixpkgs to create a branch that attempts to create a version match to specific distributions, especially Ubuntu as the ML world is most using it. My idea is to somehow use the deb package information to “shadow” another distribution. > First because Python isn't compiled with optimizations and LTO in nixpkgs by de…

I understand your comments including above and the one about CUDA binaries. Just one clarification on the concurrency in tests failure, do you mean it overloads the machine running multi process tests that then tests fail due to assumptions by the package authors?

I've found that quite some test suites have race conditions (e.g. simultaneous modification of files, etc.), which manifest themselves e.g. when a package uses pytest-xdist (and the machine has enough cores).

My main point is that Nix as a system is so incredibly powerful that perhaps there is an ability to “shadow” boring distributions,

I think things would improve vastly if it was possible to do CUDA builds in Hydra and have the resulting packages in the binary cache. My idea (when I was still contributing to nixpkgs) was to somehow mark CUDA derivations specially, so that they get built but not stored in the binary cache. That would allow packages with CUDA dependencies to get built as well (e.g. PyTorch). Nix would then only have to build CUDA locally (which is cheap, since it only entails unpacking the binary distribution and putting things in the right output paths) and would get everything else through the binary cache (like prebuilt PyTorch). But AFAIR it'd require some coordinated changes between Nix, Hydra, etc.

Then I started working for a company in the Python/Cython ecosystem and quickly found out that Nix is not really viable for most Python development. So I am now just using pyenv and pip, which works fine most of the time (we have some people in our team who are very good at maintaining proper package version bounds).

Re: A brief history of LLaMA models

#80

Post by [deleted] »

vidarh wrote: post #72
PostOnce wrote: post #54

Earlier quoted context omitted.

No need to drag a gigabyte of docker stuff into this, just extract the zip file from github and type make into your terminal congratulations, it now works. If you're not a developer, maybe you'll have to type sudo apt install build-essential first. Congratulations, now you too, a non-developer, are running it locally. https://github.com/ggerganov/llama.cpp

Did that, got a compiler error within seconds within seconds. Looks like it might need a newer version of gcc than is in the distro on my laptop. Which is why people ask for Docker. If it really will work just with make with build-essential on a new enough distro image, a Dockerfile that documents that would be trivial, and does not at all stop people from just typing make if their setup is new enough.

[deleted]
Post reply on HN