> Stick that file on a USB stick and stash it in a drawer as insurance against a future apocalypse. You’ll never be without a language model ever again.
<3
Llamafile lets you distribute and run LLMs with a single file
91–100 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#92I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Thanks for the tip! Any chance this would run on a 2011 MacBook?
do you... have any plans to upgrade? A gen 2011 computer is going to get harder and harder to make work. even a used macbook from like 2019 would probably be a steal at this point, and that's 8 years further along
All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available
Re: Llamafile lets you distribute and run LLMs with a single file
#93I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…
I get the same error, and there's no `ape` file to make excecutable, hm.
Re: Llamafile lets you distribute and run LLMs with a single file
#94I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…
You can solve the run-detectors issue with:
sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
See the Gotchas section of the README https://github.com/mozilla-Ocho/llamafile#gotchas
Re: Llamafile lets you distribute and run LLMs with a single file
#95Earlier quoted context omitted.
$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…
I get the same error, and there's no `ape` file to make excecutable, hm.
You can manually download the `ape` command from https://cosmo.zip/pub/cosmos/bin/ Please see the Gotchas section of the README for the copy/pastable commands you can run: https://github.com/mozilla-Ocho/llamafile#gotchas
Re: Llamafile lets you distribute and run LLMs with a single file
#96Earlier quoted context omitted.
> a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device that assumption gets wrong very fast with nvme storage, even before you add herding effects
Until you compare a single machine with nvme to a cluster of storage servers with nvme, and each machine has 800Gbit connectivity and you use smart replication for herding. but yes, nvme definitely has amazing transfer rates.
> Until you compare a single machine with nvme to a cluster of storage servers with nvme
No, only as long as you compare against a very low number of machines with local nvme.
The sum of the bandwith available on typical storage device (even cheap and low end) will be at most times greater than what you have of your expansive top of the line cluster
If you have a single local storage, you don't have scale, so you won't have money for an expansive top of the line cluster either. But if you are wasting money on it, yes you will have more bandwidth, but that's a degenerate case.
If you have a few local storage machines, the assumption gets very wrong and very fast: 1 low end tier nvme=1 G/s at worst, one top of the line WD 990: 8G/s at best, so we're talking about a ratio of ~ 8 in the most favorable scenario.
Re: Llamafile lets you distribute and run LLMs with a single file
#97Earlier quoted context omitted.
$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…
You can solve the run-detectors issue with: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh tw…
That worked, thanks Justine! I use fish, so I didn't get a zsh error, but I had missed the Gotchas section (and the README), so this helps!
Re: Llamafile lets you distribute and run LLMs with a single file
#98Earlier quoted context omitted.
You can solve the run-detectors issue with: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh tw…
That worked, thanks Justine! I use fish, so I didn't get a zsh error, but I had missed the Gotchas section (and the README), so this helps!
Fish is another cool shell I got to help improve two years ago by upstreaming a patch for this. So long as you're using a recent version, you should be golden (provided binfmt_misc doesn't cause any issues). Let us know what you think of llamafile!
Re: Llamafile lets you distribute and run LLMs with a single file
#99Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…
I don't get this obsession with 0-click everything. It is really annoying when you don't want to install everything to your main hard drive. I have all my models downloaded, organized, and ready-to-go but apps won't even ask for that, instead it presumes I am an idiot and downloads it (again!) for me. At least Makeayo asks where my models are now. It's obnoxious that I have to use symlinks for comfy/automatic.... All…
Sounds like you should download the 4.45MB llamafile-server-0.1 executable from https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.1 and then run it against your existing gguf model files like this:
./llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf
Re: Llamafile lets you distribute and run LLMs with a single file
#100I like the idea of putting it in one file but not an executable file. Using CBOR (MessagePack has a 4gb bytestring limit) and providing a small utility to copy the executable portion and run it would be a win. No 4gb limit. It could use delta updates.