This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.
Like a docker for LLMs
I don't see why you cannot use a container for LLMs, that's how we've shipping and deploying runnable models for years
Llamafile lets you distribute and run LLMs with a single file
111–120 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#112Earlier quoted context omitted.
Like a docker for LLMs
I don't see why you cannot use a container for LLMs, that's how we've shipping and deploying runnable models for years
Being able to run a LLM without first installing and setting up Docker or similar feels like a big win to me.
Is there an easy way to run a Docker container on macOS such that it can access the GPU?
Re: Llamafile lets you distribute and run LLMs with a single file
#113Earlier quoted context omitted.
Not at my windows machine to test this out right now, but wondering what you mean by having to store the weights in a separate file for wizardcoder, as a result of the 4gb executable limit. How does one go about this? Thank you!
You'd do something like this on PowerShell: curl -Lo llamafile.exe https://github.com/Mozilla-Ocho/llamafile/releases/download/0.1/llamafile-server-0.1 curl -Lo wizard.gguf https://huggingface.co/TheBloke/WizardCoder-Python-13B-V1.0-GGUF/resolve/main/wizardcoder-python-13b-v1.0.Q4_K_M.gguf .\llamafile.exe -m wizard.gguf
Awesome! Thank you so much
Re: Llamafile lets you distribute and run LLMs with a single file
#114Can confirm that this runs on an ancient i3 NUC under Ubuntu 20.04. It emits a token every five or six seconds, which is "ask a question then go get coffee" speed. Still, very cool.
Re: Llamafile lets you distribute and run LLMs with a single file
#115Earlier quoted context omitted.
I don't see why you cannot use a container for LLMs, that's how we've shipping and deploying runnable models for years
Being able to run a LLM without first installing and setting up Docker or similar feels like a big win to me. Is there an easy way to run a Docker container on macOS such that it can access the GPU?
Not sure, I use cloud VMs for ML stuff
We definitely prefer to use the same tech stack for dev and production, we already have docker (mostly migrated to nerdctl actually)
Can this project do production deploys to the cloud? Is it worth adding more tech to the stack for this use-case? I often wonder how much devops gets reimplemented in more specialized fields
Re: Llamafile lets you distribute and run LLMs with a single file
#116I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
so next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic.
EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again.
EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables. Such an invention.
Re: Llamafile lets you distribute and run LLMs with a single file
#117Does it use Metal on Mac OS (Apple Silicon)? And if not, how does it compare performance-wise against regular llama.cpp? It's not necessarily an advantage to pack everything (huge quantified 4bit? model and code) into a single file, or at least it wasn't when llama.cpp was gaining speed almost daily.
Re: Llamafile lets you distribute and run LLMs with a single file
#118From a technical standpoint, this project is really fascinating. I can see a lot of use cases for getting something up fast locally for an individual user. But for anyone in a production/business setting, it would be tough to see this being viable. Seems like it would be a non-starter for most medium to large companies IT teams. The great thing about a Dockerfile is that it can be inspected and the install process is…
This stuff is for people who don't care about medium to large companies IT teams.
Re: Llamafile lets you distribute and run LLMs with a single file
#119Why is this faster than running llama.cpp main directly? I’m getting 7 tokens/ sec with this. But 2 with llama.cpp by itself
Re: Llamafile lets you distribute and run LLMs with a single file
#120Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…
> I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it. Easy! But as soon as I wanted to release a UI update to my TestFlight beta testers, I was causing them to download another 3GB. All 3 us…
Yes, though it does seem to be working for them. They have a special feature for lazy loading large assets but I opted for a simpler to me option (giving users a button to download a model if they don’t have one locally they want to use).