Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

101–110 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#101

Post by rgbrgb »

pdntspa wrote: post #90
rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

I don't get this obsession with 0-click everything. It is really annoying when you don't want to install everything to your main hard drive. I have all my models downloaded, organized, and ready-to-go but apps won't even ask for that, instead it presumes I am an idiot and downloads it (again!) for me. At least Makeayo asks where my models are now. It's obnoxious that I have to use symlinks for comfy/automatic.... All…

fwiw FreeChat does this now. It prompts you to download or select a model to use (and you can add as many as you want). No copying or forced downloads.

Re: Llamafile lets you distribute and run LLMs with a single file

#102

Post by pmarreck »

mritchie712 wrote: post #37
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

woah, this is fast. On my M1 this feels about as fast as GPT-4.

Same here on M1 Max Macbook Pro. This is great!

Re: Llamafile lets you distribute and run LLMs with a single file

#103

Post by fbdab103 »

dmezzetti wrote: post #39

From a technical standpoint, this project is really fascinating. I can see a lot of use cases for getting something up fast locally for an individual user. But for anyone in a production/business setting, it would be tough to see this being viable. Seems like it would be a non-starter for most medium to large companies IT teams. The great thing about a Dockerfile is that it can be inspected and the install process is…

I am failing to see the difference. It is a zip file with an executable and a blob of weights. What would change if it were stored in a Dockerfile?

Re: Llamafile lets you distribute and run LLMs with a single file

#104

Post by pmarreck »

jart wrote: post #94
lol768 wrote: post #62

Earlier quoted context omitted.

$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…

You can solve the run-detectors issue with: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh tw…

Yet another jart tour-de-force. I knew I had to sponsor you on Github back when I read your magnificent technical breakdown of APE, lol.

(sorry for OT!)

Re: Llamafile lets you distribute and run LLMs with a single file

#105

Post by tatrajim »

Small field test: I uploaded a picture of a typical small Korean Buddhist temple, with a stone pagoda in front. Anyone at all familiar with East Asian Buddhism would instantly recognize both the pagoda and the temple behind it as Korean.

Llamafile: "The image features a tall, stone-like structure with many levels and carved designs on it. It is situated in front of an Asian temple building that has several windows. In the vicinity, there are two cars parked nearby – one closer to the left side of the scene and another further back towards the right edge. . ."

ChatGPT4:"The photo depicts a traditional Korean stone pagoda, exhibiting a tiered tower with multiple levels, each diminishing in size as they ascend. It is an example of East Asian pagodas, which are commonly found within the precincts of Buddhist temples. . . The building is painted in vibrant colors, typical of Korean temples, with green being prominent."

No comparison, alas.

Re: Llamafile lets you distribute and run LLMs with a single file

#106

Post by jart »

Hedepig wrote: post #22
estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

I am currently tinkering with this all, you can download a 3b parameter model and run it on your phone. Of course it isn't that great, but I had a 3b param model[1] on my potato computer (a mid ryzen cpu with onboard graphics) that does surprisingly well on benchmarks and my experience has been pretty good with it. Of course, more interesting things happen when you get to 32b and the 70b param models, which will requ…

That's a nice model that fits comfortably on Raspberry Pi. It's also only a few days old! I've just finished cherry-picking the StableLM support from the llama.cpp project upstream that you'll need in order to run these weights using llamafile. Enjoy! https://github.com/Mozilla-Ocho/llamafile/commit/865462fc465...

Re: Llamafile lets you distribute and run LLMs with a single file

#107

Post by askiiart »

RecycledEle wrote: post #57

Fantastic. For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a huge number of dependencies? It seems like someone would have included Llama.cpp in their distro, ready-to-run. Yes, I'm an idiot.

> It seems like someone would have included Llama.cpp in their distro, ready-to-run.

Assuming you mean installable with a package manager, not preinstalled on a distro, that requires that some maintainer decide it's worthwhile to add it and maintain it. Distros are pretty selective in what they add to their repos, but there's probably a tool for building .deb or .rpm packages of llama.cpp, and probably a repository for it, but as far as I know no distro has llama.cpp in its repos.

Or Arch Linux's AUR system is much more open, and it indeed has llama-cpp (4 versions of it!), though it requires a helper, such as yay, if you want to install it and keep it up-to-date as if it were a normal package. So Arch has it installable with a package manager if you use yay to supplement pacman.

https://aur.archlinux.org/packages?O=0&K=llama-cpp

Re: Llamafile lets you distribute and run LLMs with a single file

#108

Post by jmorgan »

insanitybit wrote: post #70
andersa wrote: post #40

Earlier quoted context omitted.

They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. The reason for this flag to exist in the first place is that many of the models are larger than the available VRAM on most consumer GPUs, so you have to "balance" it between running some layers on the GPU and some on the CPU. What wo…

> They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. What I don't understand is why it can't just check your VRAM and allocate by default. The allocation is not that dynamic AFAIK - when I run models it all happens basically upfront when the model loads. ollama even prints out how mu…

This is a great point. Context size has a large impact on memory requirements and Ollama should take this into account (something to work on :)

Re: Llamafile lets you distribute and run LLMs with a single file

#109

Post by simonw »

tatrajim wrote: post #105

Small field test: I uploaded a picture of a typical small Korean Buddhist temple, with a stone pagoda in front. Anyone at all familiar with East Asian Buddhism would instantly recognize both the pagoda and the temple behind it as Korean. Llamafile: "The image features a tall, stone-like structure with many levels and carved designs on it. It is situated in front of an Asian temple building that has several windows. I…

That's not a llamafile thing, that's a llava-v1.5-7b-q4 thing - you're running the LLaVA 1.5 model at a 7 billion parameter size further quantized to 4 bits (the q4).

GPT4-Vision is running a MUCH larger model than the tiny 7B 4GB LLaVA file in this example.

LLaVA have a 13B model available which might do better, though there's no chance it will be anywhere near as good as GPT-4 Vision. https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZO...

Post reply on HN