31–40 of 302 posts
I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.
Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…
I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.
There's also a "llamafile" 4MB binary that can run any model (GGUF file) that you pass to it: https://simonwillison.net/2023/Nov/29/llamafile/#llamafile-t...
I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".
Earlier quoted context omitted.
There's also a "llamafile" 4MB binary that can run any model (GGUF file) that you pass to it: https://simonwillison.net/2023/Nov/29/llamafile/#llamafile-t...
Right. So if that exists, why would I want to embed my weights in the binary rather than distributing them as a side file? I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".
If the weights are 4GB, and the binary code needed to actually execute them is 4.5MB, then the size of the executable part is a rounding error - I don't see any reason NOT to bundle that with the model.
I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Earlier quoted context omitted.
There's also a "llamafile" 4MB binary that can run any model (GGUF file) that you pass to it: https://simonwillison.net/2023/Nov/29/llamafile/#llamafile-t...
Right. So if that exists, why would I want to embed my weights in the binary rather than distributing them as a side file? I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".
But for anyone in a production/business setting, it would be tough to see this being viable. Seems like it would be a non-starter for most medium to large companies IT teams. The great thing about a Dockerfile is that it can be inspected and the install process is relatively easy to understand.
Earlier quoted context omitted.
That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?
This is a bit like saying if you don't specify "--dram", the data will be stored on punchcards. From the user's point of view: they just want to run the thing, and as quickly as possible. If multiple programs want to use the GPU, then the OS and/or the driver should figure it out.
The reason for this flag to exist in the first place is that many of the models are larger than the available VRAM on most consumer GPUs, so you have to "balance" it between running some layers on the GPU and some on the CPU.
What would make sense is a default auto option that uses as much VRAM as possible, assuming the model is the only thing running on the GPU, except for the amount of VRAM already in use at the time it is started.