Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

181–190 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#181

Post by pdntspa »

mft_ wrote: post #149
pdntspa wrote: post #90

Earlier quoted context omitted.

I don't get this obsession with 0-click everything. It is really annoying when you don't want to install everything to your main hard drive. I have all my models downloaded, organized, and ready-to-go but apps won't even ask for that, instead it presumes I am an idiot and downloads it (again!) for me. At least Makeayo asks where my models are now. It's obnoxious that I have to use symlinks for comfy/automatic.... All…

If I'm understanding (and agreeing with) your gripe correctly, isn't it two solutions to the same perceived problem? My experience is that the world of Python dependency management is a mess which sometimes works, and sometimes forces you to spend hours-to-days searching for obscure error messages and trying maybe-fixes posted in Github issues for some other package, just in case it helps. This sometimes extends furt…

I get that, my issue is when the model is coupled with the app, or the app just presumes I don't have it downloaded and doesn't ask me otherwise. This is like basic configuration stuff...

What I suspect is happening is that people are cargo-culting zero-click installations. It seems rather fashionable right now.

Re: Llamafile lets you distribute and run LLMs with a single file

#182

Post by creshal »

botanical wrote: post #178
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

I just tried asking it a question: > User: What is the third planet from the sun? > Llama: The third planet from the sun is called Mars.

Skynet is collaborating with the Martians already, I see.

Re: Llamafile lets you distribute and run LLMs with a single file

#183

Post by wongarsu »

numpad0 wrote: post #130
insanitybit wrote: post #70

Earlier quoted context omitted.

> They don't, though. If you try to allocate too much VRAM it will either hard fail or everything suddenly runs like garbage due to the driver constantly swapping it / using shared memory. What I don't understand is why it can't just check your VRAM and allocate by default. The allocation is not that dynamic AFAIK - when I run models it all happens basically upfront when the model loads. ollama even prints out how mu…

Some GPUs has quirks that VRAM access slows down near the end or that GPU just crashes and disables display output if actually used. I think it's sort of sensible that they don't use GPU at all by default.

Wouldn't the sensible default be to use 80% of available VRAM, or total VRAM minus 2GB, or something along those lines. Something that's a tad conservative but works for 99% of cases, with tuning options for those who want to fly closer to the sun.

Re: Llamafile lets you distribute and run LLMs with a single file

#184

Post by abareplace »

jart wrote: post #18
brucethemoose2 wrote: post #7

Earlier quoted context omitted.

> On Windows, that usually means you need to open up the MSVC x64 native command prompt and run llamafile there, for the first invocation, so it can build a DLL with native GPU support. After that, $CUDA_PATH/bin still usually needs to be on the $PATH so the GGML DLL can find its other CUDA dependencies. Yeah, I think the setup lost most users there. A separate model/app approach (like Koboldcpp) seems way easier TBH…

Author here. llamafile will work on stock Windows installs using CPU inference. No CUDA or MSVC or DLLs are required! The dev tools are only required to be installed, right now, if you want get faster GPU performance.

The CPU usage is around 30% when idle (not handling any HTTP requests) under Windows, so you won't want to keep this app running in background. Otherwise, it's a nice try.

Re: Llamafile lets you distribute and run LLMs with a single file

#185

Post by tmountain »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

This is amazing. How does result quality compare to GPT4 for image analysis?

Re: Llamafile lets you distribute and run LLMs with a single file

#186

Post by reisse »

finnjohnsen2 wrote: post #168
reisse wrote: post #165

Earlier quoted context omitted.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.

I see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume for model?

Cargo cult is not in using Docker, but in using Docker to wrap something already wrapped into a comparable layer of abstraction.

Besides,

> not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet

is purely self-deception. It's not like Docker images are not stored in some folder deep in the filesystem. If anything, it's harder to clean up after Docker than just doing rm -rf on a directory with llamafiles.

Re: Llamafile lets you distribute and run LLMs with a single file

#187

Post by reisse »

tluyben2 wrote: post #175
reisse wrote: post #165

Earlier quoted context omitted.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

Personally I wanted to see if this llama can generate the required docker files to run itself just from pasting in the instructions from the my parent commenter and asking for docker/docker-compose. It did and it works.

Well that's a good reason I didn't think about, thank you!

Re: Llamafile lets you distribute and run LLMs with a single file

#188

Post by larodi »

rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

Perhaps another unpopular opinion that can get the comment outright down-voted, but still... While jart's work is very interesting in nature and execution, commendable stuff indeed of a person with very high IQ and discipline, I still wonder whether Justine simply can't get over the fact they got kicked out of the llama.cpp project (yes, I understand jart is frequenting HN, and also let's agree llama.cpp is at least as cool as jart's projcets). No, I'm not going in details of said dismissal, as both sides seem to have had their proper arguments, but still.

And of course, I can imagine where the whole cosmopolitan thing comes from,... even as manifest of sorts for the idea of systems-neutrality and potentially gender fluidity. But I really wonder whether GGUF actually needs this, since llama.cpp already compiles and runs pretty much everywhere.

Why introduce one more container? Who benefits from binary distribution of this sort?

Re: Llamafile lets you distribute and run LLMs with a single file

#189

Post by epaga »

Hmmm no matter what image I upload, it's somehow not seeing it.

Got a chuckle out of this, though: ---

User: Describe the image in the most detailed way possible

Llama: The image is a close-up of an empty, dark room with no visible objects or people. It appears to be a black background photo that highlights the darkness and emptiness of the space. This could potentially represent an artistic representation or an actual setting where lighting has been used to emphasize the contrast between light and shadow. The focus on this particular area suggests that it might have some significance, whether symbolic or practical in nature. Overall, the image evokes a sense of mystery and intrigue due to its lack of context and content.

User: is it just a black square?

Llama: Yes, it is just a black square.

Re: Llamafile lets you distribute and run LLMs with a single file

#190

Post by m1thrandir »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Is the process the same for running multi-part bins? Like the latest deekseek 67b model?
Post reply on HN