Currently which are the minimum system requirements for running these models?
Basically enough to fit the download in RAM + a bit more.
In practice, you kinda need a GPU, even a small one. Otherwise prompt processing is really slow.
Llamafile lets you distribute and run LLMs with a single file
61–70 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#62I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4
$ ./llamafile-server-0.1-llava-v1.5-7b-q4
run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4
Installing the portable binfmt_misc gets me further, but still:
$ ./llamafile-server-0.1-llava-v1.5-7b-q4
zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4
$ sh -c ./llamafile-server-0.1-llava-v1.5-7b-q4
sh: 1: ./llamafile-server-0.1-llava-v1.5-7b-q4: Permission deniedRe: Llamafile lets you distribute and run LLMs with a single file
#63> you pass the --n-gpu-layers 35 flag (or whatever value is appropriate) to enable GPU This is a bit like specifying how large your strings will be to a C program. That was maybe accepted in the old days, but not anymore really.
That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?
> What other API would you suggest?
MLC LLM?
I think the binary it compiles down to (Probably the Vulkan and Metal ones for yall) is seperate from the weights, so you could ship a bunch in one file.
Re: Llamafile lets you distribute and run LLMs with a single file
#64This is not to be dismissive but there is a security risk if we keep on using the abstraction with arbitrary objects being serialized to disk and being able to trace back and see if the model file (most commonly python pickle files) aren’t tampered with .
You just need to have a stray TXT file in your system, or even downloaded from internet that prompts the AI to hack your system. If your AI has Python sand box and that has vulnerabilities, you can be hacked by any web page or text file. And the AI would be able to study your computer and select the most juicy bits to send out. It would be like a sentient virus spread by simple text files (text bombs?).
[deleted]
Re: Llamafile lets you distribute and run LLMs with a single file
#65I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Super duper impressed. I've run llamafile-server-0.1-llava-v1.5-7b-q4 against the tests I need to pass for use in a project, and this passes them all, vision queries too. This is gonna change quite a bit, strategy-wise for quite a few people.
Re: Llamafile lets you distribute and run LLMs with a single file
#66Earlier quoted context omitted.
Right. So if that exists, why would I want to embed my weights in the binary rather than distributing them as a side file? I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".
This is convenient for people who don't want to go knee deep in LLM-ology to try an LLM out on their computer. That said a single download that in turn downloads the weights for you is just as good in my book.
`ollama pull ` has worked for me, and then I can try out new models and updated the binary trivially.
Re: Llamafile lets you distribute and run LLMs with a single file
#67Earlier quoted context omitted.
Thanks for the tip! Any chance this would run on a 2011 MacBook?
Justine says it needs MacOS 13.6+ - does that run on that machine?
Yes, with a patch https://en.wikipedia.org/wiki/MacBook_Pro#macOS
from https://dortania.github.io/OpenCore-Legacy-Patcher/MODELS.ht...
I thought my 2015 MBP wasn't able to upgrade. Good to know it's still supported.
Re: Llamafile lets you distribute and run LLMs with a single file
#68This is not to be dismissive but there is a security risk if we keep on using the abstraction with arbitrary objects being serialized to disk and being able to trace back and see if the model file (most commonly python pickle files) aren’t tampered with .
You just need to have a stray TXT file in your system, or even downloaded from internet that prompts the AI to hack your system. If your AI has Python sand box and that has vulnerabilities, you can be hacked by any web page or text file. And the AI would be able to study your computer and select the most juicy bits to send out. It would be like a sentient virus spread by simple text files (text bombs?).
This sounds very implausible. Is there like a proof of concept I missed?
Re: Llamafile lets you distribute and run LLMs with a single file
#69Earlier quoted context omitted.
You just need to have a stray TXT file in your system, or even downloaded from internet that prompts the AI to hack your system. If your AI has Python sand box and that has vulnerabilities, you can be hacked by any web page or text file. And the AI would be able to study your computer and select the most juicy bits to send out. It would be like a sentient virus spread by simple text files (text bombs?).
This sounds very implausible. Is there like a proof of concept I missed?
The bible. Have you heard the good word of Jesus Christ?
[It's not sentient by itself but it's a self-replicating memeplex that activates in a "mind"]
Re: Llamafile lets you distribute and run LLMs with a single file
#70Earlier quoted context omitted.
This is a bit like saying if you don't specify "--dram", the data will be stored on punchcards. From the user's point of view: they just want to run the thing, and as quickly as possible. If multiple programs want to use the GPU, then the OS and/or the driver should figure it out.
They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. The reason for this flag to exist in the first place is that many of the models are larger than the available VRAM on most consumer GPUs, so you have to "balance" it between running some layers on the GPU and some on the CPU. What wo…
> They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory.
What I don't understand is why it can't just check your VRAM and allocate by default. The allocation is not that dynamic AFAIK - when I run models it all happens basically upfront when the model loads. ollama even prints out how much VRAM it's allocating for model + context for each layer. But I still have to tune the layers manually, and any time I change my context size I have to retune.