> Windows also has a maximum file size limit of 2GB for executables. You need to have llamafile and your weights be separate files on the Windows platform. The 4GB .exe ran fine on my Windows 10 64-bit system.
[deleted]
Llamafile lets you distribute and run LLMs with a single file
161–170 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#162I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
so next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic. EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again. EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables.…
salty much?
You know, most people don't have 24+GB GPUs sitting around to train these models. So in my book this is a huge step forward. Personally, this is the first time i am able to run an LLM on my computer, and it's purely thanks to this.
Re: Llamafile lets you distribute and run LLMs with a single file
#163I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
It was already great, and this is more great for those who get Docker-Compose or are patient enough to figure out. But if you're gonna have Docker, you could also use bleeding edge llama.cpp with a few more lines too! What a time to be alive innit!
Re: Llamafile lets you distribute and run LLMs with a single file
#164I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
woah, this is fast. On my M1 this feels about as fast as GPT-4.
How good is it in comparison
Re: Llamafile lets you distribute and run LLMs with a single file
#165I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question?
Looks like cargo cult for me.
Re: Llamafile lets you distribute and run LLMs with a single file
#166I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
Thank you kindly
Re: Llamafile lets you distribute and run LLMs with a single file
#167Earlier quoted context omitted.
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
Deploy your models into a Kubernetes cluster and let them fight for resources to death? A modern Roman Gladiators circus with Models?
Re: Llamafile lets you distribute and run LLMs with a single file
#168Earlier quoted context omitted.
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.
Re: Llamafile lets you distribute and run LLMs with a single file
#169Earlier quoted context omitted.
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
Wrapping it in Docker makes it harder to access your filesystem without some dedicated jailbreak.
Re: Llamafile lets you distribute and run LLMs with a single file
#170Im sure this is great, but not screenshot of the GUI?
Simon Willison's blog post has a screenshot. It's worth a read. https://simonwillison.net/2023/Nov/29/llamafile/
Thank you, is there a way to select a different model? How does the model perform? Just general question if anyone else knows the answers while I try and clear space on my laptop ( why these things fill up so fast!)