Llamafile lets you distribute and run LLMs with a single file
1–10 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#2This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.
Re: Llamafile lets you distribute and run LLMs with a single file
#3This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.
yeah the section on how the GPU support works is wild!
Re: Llamafile lets you distribute and run LLMs with a single file
#4The technical details in the README are quite an interesting read:
Re: Llamafile lets you distribute and run LLMs with a single file
#5Related: https://hacks.mozilla.org/2023/11/introducing-llamafile/ and https://twitter.com/justinetunney/status/1729940628098969799
(via https://news.ycombinator.com/item?id=38463456 and https://news.ycombinator.com/item?id=38464759, but we merged the comments hither)
Re: Llamafile lets you distribute and run LLMs with a single file
#6Re: Llamafile lets you distribute and run LLMs with a single file
#7Earlier quoted context omitted.
yeah the section on how the GPU support works is wild!
So if you share a binary with a friend you'd have to have them install cuda toolkit too? Seems like a dealbreaker for the whole idea.
> On Windows, that usually means you need to open up the MSVC x64 native command prompt and run llamafile there, for the first invocation, so it can build a DLL with native GPU support. After that, $CUDA_PATH/bin still usually needs to be on the $PATH so the GGML DLL can find its other CUDA dependencies.
Yeah, I think the setup lost most users there.
A separate model/app approach (like Koboldcpp) seems way easier TBH.
Also, GPU support is assumed to be CUDA or Metal.
Re: Llamafile lets you distribute and run LLMs with a single file
#8Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one.
I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it. Easy! But as soon as I wanted to release a UI update to my TestFlight beta testers, I was causing them to download another 3GB. All 3 users complained :). My first change after that was decoupling the default model download and the UI so that I can ship app updates that are about 5MB. It feels like someone using this tool is going to hit the same problem pretty quick when they want to get the latest llama.cpp updates (ggerganov SHIIIIPS [2]). Maybe there are cases where that doesn't matter, would love to hear where people think this could be useful.
Re: Llamafile lets you distribute and run LLMs with a single file
#9Earlier quoted context omitted.
So if you share a binary with a friend you'd have to have them install cuda toolkit too? Seems like a dealbreaker for the whole idea.
> On Windows, that usually means you need to open up the MSVC x64 native command prompt and run llamafile there, for the first invocation, so it can build a DLL with native GPU support. After that, $CUDA_PATH/bin still usually needs to be on the $PATH so the GGML DLL can find its other CUDA dependencies. Yeah, I think the setup lost most users there. A separate model/app approach (like Koboldcpp) seems way easier TBH…
[deleted]
Re: Llamafile lets you distribute and run LLMs with a single file
#10Earlier quoted context omitted.
So if you share a binary with a friend you'd have to have them install cuda toolkit too? Seems like a dealbreaker for the whole idea.
> On Windows, that usually means you need to open up the MSVC x64 native command prompt and run llamafile there, for the first invocation, so it can build a DLL with native GPU support. After that, $CUDA_PATH/bin still usually needs to be on the $PATH so the GGML DLL can find its other CUDA dependencies. Yeah, I think the setup lost most users there. A separate model/app approach (like Koboldcpp) seems way easier TBH…
I'm sure doing better by windows users is on the roadmap, exec then reexec to get into the right runtime, but it's a good first step towards making things easy.