This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.
yeah the section on how the GPU support works is wild!
Why don't package managers do stuff like this?
Llamafile lets you distribute and run LLMs with a single file
11–20 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#12> you pass the --n-gpu-layers 35 flag (or whatever value is appropriate) to enable GPU
This is a bit like specifying how large your strings will be to a C program. That was maybe accepted in the old days, but not anymore really.
Re: Llamafile lets you distribute and run LLMs with a single file
#13Simon Willison has a great post on this https://simonwillison.net/2023/Nov/29/llamafile/
Re: Llamafile lets you distribute and run LLMs with a single file
#14I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though):
1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/...:
wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llamafile-server-0.1-llava-v1.5-7b-q4
chmod 755 llamafile-server-0.1-llava-v1.5-7b-q4
./llamafile-server-0.1-llava-v1.5-7b-q4
Screenshot here: https://simonwillison.net/2023/Nov/29/llamafile/
Re: Llamafile lets you distribute and run LLMs with a single file
#15> you pass the --n-gpu-layers 35 flag (or whatever value is appropriate) to enable GPU This is a bit like specifying how large your strings will be to a C program. That was maybe accepted in the old days, but not anymore really.
That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?
Re: Llamafile lets you distribute and run LLMs with a single file
#16Currently which are the minimum system requirements for running these models?
Re: Llamafile lets you distribute and run LLMs with a single file
#17[deleted]
Re: Llamafile lets you distribute and run LLMs with a single file
#18Earlier quoted context omitted.
So if you share a binary with a friend you'd have to have them install cuda toolkit too? Seems like a dealbreaker for the whole idea.
> On Windows, that usually means you need to open up the MSVC x64 native command prompt and run llamafile there, for the first invocation, so it can build a DLL with native GPU support. After that, $CUDA_PATH/bin still usually needs to be on the $PATH so the GGML DLL can find its other CUDA dependencies. Yeah, I think the setup lost most users there. A separate model/app approach (like Koboldcpp) seems way easier TBH…
Author here. llamafile will work on stock Windows installs using CPU inference. No CUDA or MSVC or DLLs are required! The dev tools are only required to be installed, right now, if you want get faster GPU performance.
Re: Llamafile lets you distribute and run LLMs with a single file
#19A couple of steps away from getting weaponized.
Re: Llamafile lets you distribute and run LLMs with a single file
#20This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.
Like a docker for LLMs