Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

121–130 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#121

Post by Zuiii »

jxy wrote: post #116
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

so next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic. EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again. EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables.…

That's why I always download the original version and quantize myself. With enough swap, you can do it with a modest amount for ram. I never had to download a model twice.

But yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.

Re: Llamafile lets you distribute and run LLMs with a single file

#123

Post by thejosh »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Damn this is fast and accurate! Crazy how far things are progressing.

Re: Llamafile lets you distribute and run LLMs with a single file

#124

Post by wyldfire »

rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

> Extremely cool ...

> I'm kind of struggling with the use-cases for this one.

IMO cosmopolitan libc is a "really neat trick". And it deserves praise and it probably does have some real use cases. But it's not practical for most purposes. If we had a format like ELF that was so fat as to support as many architectures and OSs as desired, would we be using that? I have a feeling that we would not.

Then again -- after having used "zig cc" for a while, maybe it would be reasonable to have something like "one build" that produces a mega-fat binary.

And the microarch-specific dispatch is a nice touch.

...maybe I'm convincing myself of the alternative....

Re: Llamafile lets you distribute and run LLMs with a single file

#125

Post by insanitybit »

jmorgan wrote: post #108
insanitybit wrote: post #70

Earlier quoted context omitted.

> They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. What I don't understand is why it can't just check your VRAM and allocate by default. The allocation is not that dynamic AFAIK - when I run models it all happens basically upfront when the model loads. ollama even prints out how mu…

This is a great point. Context size has a large impact on memory requirements and Ollama should take this into account (something to work on :)

Thanks for the work you've done already :D

Re: Llamafile lets you distribute and run LLMs with a single file

#126

Post by simonw »

jxy wrote: post #116
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

so next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic. EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again. EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables.…

Each llamafile is a .zip, so if you want to extract the weights out of it you can extract the gguf file directly.

    unzip -l llamafile-server-0.1-llava-v1.5-7b-q4 | grep llava-v1
    Archive:  llamafile-server-0.1-llava-v1.5-7b-q4
    4081004224  11-15-2023 22:13   llava-v1.5-7b-Q4_K.gguf
    177415936  11-15-2023 22:13   llava-v1.5-7b-mmproj-Q4_0.gguf

Re: Llamafile lets you distribute and run LLMs with a single file

#127

Post by simonw »

tannhaeuser wrote: post #117

Does it use Metal on Mac OS (Apple Silicon)? And if not, how does it compare performance-wise against regular llama.cpp? It's not necessarily an advantage to pack everything (huge quantified 4bit? model and code) into a single file, or at least it wasn't when llama.cpp was gaining speed almost daily.

It uses the GPU on my M2 Mac - I can see it making use of that in the Activity Monitor GPU panel.

Re: Llamafile lets you distribute and run LLMs with a single file

#128

Post by jart »

simonw wrote: post #127
tannhaeuser wrote: post #117

Does it use Metal on Mac OS (Apple Silicon)? And if not, how does it compare performance-wise against regular llama.cpp? It's not necessarily an advantage to pack everything (huge quantified 4bit? model and code) into a single file, or at least it wasn't when llama.cpp was gaining speed almost daily.

It uses the GPU on my M2 Mac - I can see it making use of that in the Activity Monitor GPU panel.

Correct. Apple Silicon GPU performance should be equally fast in llamafile as it is in llama.cpp. Where llamafile is currently behind is at CPU inference (only on Apple Silicon specifically) which is currently going ~22% slower compared to a native build of llama.cpp. I suspect it's due to either (1) I haven't implemented support for Apple Accelerate yet, or (2) our GCC -march=armv8a toolchain isn't as good at optimizing ggml-quant.c as Xcode clang -march=native is. I hope it's an issue we can figure out soon!

Re: Llamafile lets you distribute and run LLMs with a single file

#129

Post by aphroz »

jxy wrote: post #116
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

so next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic. EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again. EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables.…

Compared to modern bandwidth usage that's not such a big size anymore. Everyday millions of people download 100gb video games, watch 4k video podcasts, etc.

Re: Llamafile lets you distribute and run LLMs with a single file

#130

Post by numpad0 »

insanitybit wrote: post #70
andersa wrote: post #40

Earlier quoted context omitted.

They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. The reason for this flag to exist in the first place is that many of the models are larger than the available VRAM on most consumer GPUs, so you have to "balance" it between running some layers on the GPU and some on the CPU. What wo…

> They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. What I don't understand is why it can't just check your VRAM and allocate by default. The allocation is not that dynamic AFAIK - when I run models it all happens basically upfront when the model loads. ollama even prints out how mu…

Some GPUs has quirks that VRAM access slows down near the end or that GPU just crashes and disables display output if actually used. I think it's sort of sensible that they don't use GPU at all by default.
Post reply on HN