Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history of LLaMA models

agi-sphere.com

Post reply on HN
Print view View original

41–50 of 87 posts

Re: A brief history of LLaMA models

#41

Post by AnthonyMouse »

spudlyo wrote: post #28
AnthonyMouse wrote: post #21

Earlier quoted context omitted.

Most of these implementations are not platform-specific. I've been running llama.cpp on x86_64 hardware and the performance is fine. The small models are fast and the quantized 65B model generates about a token per second on a system with dual-channel DDR4, which isn't unusable. The tough thing to find is something affordable that will run the unquantized 65B model at an acceptable speed. You can put 128GB of RAM in…

If you want to spend $4800.00 on just the computer, you can get a Mac Studio with 128G of memory with 400GB/s bandwidth. There are sparse reports out there of folks running 65B models on it. I've seen no performance measurements though.

It's interesting that they actually have it but the price is still silly.

  SP5 system board ~$1000
  Epyc 9124 $1083
  192GB registered DDR5 (12x16GB) ~$1000
  case, power supply, modest storage: ~$300
460GB/s bandwidth from 12 memory channels, 50% more memory and you'd have more than $1000 left over. But >$3000 is not a low price either, it's just lower.

Re: A brief history of LLaMA models

#42

Post by DustinBrett »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

For me it's because most of what I am learning and trying to do is applicable to LLM's in general. One day the right model will come along, until then I want to play.

Re: A brief history of LLaMA models

#43

Post by avereveard »

jiggawatts wrote: post #34
valine wrote: post #26

Earlier quoted context omitted.

I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.

> the trick is to search the project’s github issue tracker. What ever happened to the crazy notion of Dockerfiles that simply build successfully? Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?

The problem with that is gpu access from the docker image and iirc Nvidia doesn't have a Windows passtrough cuda driver

Re: A brief history of LLaMA models

#44

Post by nullc »

brianjking wrote: post #29

I'll never understand why everyone is spending so much time on a model you cannot use commercially (at all). Secondly, most of us can't even use the model for research or personal use, given the license.

The notion that model weights are copyrightable is absurd on its face. In the US you cannot gain a copyright though sweat of the brow, there must be substantial creative work. Nor does mere collection (e.g. feist v rural) create a copyright. Feeding common crawl to a standard network structure and letting an optimizer do its thing isn't creative, it's just expensive. It requires expertise and skill, sure but so does creating a phone book.

The companies working on AI would be foolish to argue for more copyrightability are because it would be hard to conclude the models were copyrightable works without also concluding that the models are unlawful derivatives of the material they were trained on. "Congrats, models can be owned, but regrets: you're bankrupt now because you just committed 4.6 billion acts of copyright infringement carrying statutory damages of $250k each."

You might argue that this is far from sure, OKAY-- but parties that take this view will out-compete ones that don't. If it does turn out to be problematic, the people that had something to work from now will pivot to backing their work on something else and will still be ahead of people sitting on their hands.

You could see it as a calculated risk, but it seems at least as safe as the one behind the underlying authors of the model weights training on material they're not licensed to distribute.

Re: A brief history of LLaMA models

#45

Post by dagenix »

jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

There is a lot that can be improved with python packaging, but calling it "childish" is itself a pretty immature comment.

Re: A brief history of LLaMA models

#46

Post by celdon25 »

UncleEntity wrote: post #38
jiggawatts wrote: post #34

Earlier quoted context omitted.

> the trick is to search the project’s github issue tracker. What ever happened to the crazy notion of Dockerfiles that simply build successfully? Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?

They require someone to take time out of their busy schedule to build? Complaining that people won’t work for you for free is a bit much, don’t you think?

I remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community).

Now it is apparently seen as "working for free for ungrateful people"

Re: A brief history of LLaMA models

#47

Post by esquire_900 »

hewlett wrote: post #32
api wrote: post #22

Earlier quoted context omitted.

A lot of people just stick cloudflare in front of anything because of cargo cultism. A $5/mo VPS can serve a blog to tens of thousands of people unless you are running something stupidly inefficient. If it’s a static blog make that hundreds of thousands. For millions you might need to splurge on the $10 or $20 per month VPS.

You can either spend $5 per month for VPS for a webserver for your static blog which you now have to secure properly, or you can just stick it on Cloudflare Pages for free

Cloudflare bot protection adds a slight delay (as in seconds) at best, and completely blocks users like the parent comment at worst. It costs no money, but isn't free either.

Re: A brief history of LLaMA models

#48

Post by valine »

celdon25 wrote: post #46
UncleEntity wrote: post #38

Earlier quoted context omitted.

They require someone to take time out of their busy schedule to build? Complaining that people won’t work for you for free is a bit much, don’t you think?

I remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community). Now it is apparently seen as "working for free for ungrateful people"

Most of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?

Re: A brief history of LLaMA models

#49

Post by jiggawatts »

dagenix wrote: post #45
jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

There is a lot that can be improved with python packaging, but calling it "childish" is itself a pretty immature comment.

Is it?

Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence.

E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68...

Take this line:

    pip3 install torch torchvision torchaudio
Which version of torch is this? The latest.

    FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04
Which version of CUDA is this? An incompatible one, apparently. Game over.

Check out "requirements.txt":

    accelerate==0.18.0
    colorama
    datasets
    flexgen==0.1.7
    gradio==3.25.0
    markdown
    numpy
    pandas
    Pillow>=9.5.0
    pyyaml
    requests
    rwkv==0.7.3
    safetensors==0.3.0
    sentencepiece
    tqdm
Wow. Less than half of those have any version specified. The rest? "Meh, I don't care, whatever."

Then this beauty:

    git+https://github.com/huggingface/peft
I love reaching out to the Internet in the middle of a build pipeline to pull the latest commit of a random repo, because that's so nice and safe, scalable, and cacheable in an artefact repository!

The NPM ecosystem gets regularly excoriated for the exact same mistakes, which by now are so well known, so often warned against, so often exploited, so regularly broken that it's getting boring.

It's like SQL injection. If you're still doing it in 2023, if your site is still getting hacked because of it, then you absolutely deserve to be labelled immature and even childish.

Re: A brief history of LLaMA models

#50

Post by jiggawatts »

KETpXDDzR wrote: post #11
jiggawatts wrote: post #10

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, bo…

llama.cpp was easy to setup IMO

Can you link to a working Dockerfile?

I've heard several people say that it is easy, but then surely it ought to be trivial to set script the build so that it works reliable in a container!

Post reply on HN