Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

41–50 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#41
post #12

> you pass the --n-gpu-layers 35 flag (or whatever value is appropriate) to enable GPU This is a bit like specifying how large your strings will be to a C program. That was maybe accepted in the old days, but not anymore really.

That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?

> What other API would you suggest?

Assuming increasing vram leads to an appreciable improvement in model speed, it should default to using all but 10% of the vram of the largest GPU, or all but 1GB, whichever is less.

If I've got 8GB of vram, the software should figure out the right number of layers to offload and a sensible context size, to not exceed 7GB of vram.

(Although I realise the authors are just doing what llama.cpp does, so they didn't design it the way it is)

Re: Llamafile lets you distribute and run LLMs with a single file

#42
post #36
post #35

Earlier quoted context omitted.

Right. So if that exists, why would I want to embed my weights in the binary rather than distributing them as a side file? I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".

Personally I really like the single file approach. If the weights are 4GB, and the binary code needed to actually execute them is 4.5MB, then the size of the executable part is a rounding error - I don't see any reason NOT to bundle that with the model.

I guess in every world I've worked in, deployment involved deploying a small executable which would run millions of times on thousands of servers, each instance loading a different model (or models) over its lifetime, and the weights are stored in a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device. The executable itself doesn't even contain the final model- just a description of the model which is compiled only after the executable starts (so the compilation has all the runtime info on the machine it will run on).

But, I think llama plus obese binaries must be targeting a very, very different community- one that doesn't build its own binaries, runs in any number of different locations, and focuses on getting the model to run with the least friction.

Re: Llamafile lets you distribute and run LLMs with a single file

#43
post #30

> Windows also has a maximum file size limit of 2GB for executables. You need to have llamafile and your weights be separate files on the Windows platform. The 4GB .exe ran fine on my Windows 10 64-bit system.

You're right. The limit is 4 gibibytes. Astonishingly enough, the llava-v1.5-7b-q4-server.llamafile is 0xfe1c0ed4 bytes in size, which is just 30MB shy of that limit. https://github.com/Mozilla-Ocho/llamafile/commit/81c6ad3251f...

Re: Llamafile lets you distribute and run LLMs with a single file

#44
post #26

I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.

I understand the feeling. It may be caused by habit rather than objectivity, though. Those open-source AI hacks are undergoing early productization: while they were only research, their modularity mattered for experimentization, but as they get closer to something that can ship, the one-click binary form factor is a nice stepping stone.

It is similar in my mind to the early days of Linux, where you had to compile it yourself and tweaked some compiler flags, compared to now, where most people don’t even think about the fact that their phone or Steam deck runs it.

Re: Llamafile lets you distribute and run LLMs with a single file

#46
This is not to be dismissive but there is a security risk if we keep on using the abstraction with arbitrary objects being serialized to disk and being able to trace back and see if the model file (most commonly python pickle files) aren’t tampered with .

Re: Llamafile lets you distribute and run LLMs with a single file

#48

This is not to be dismissive but there is a security risk if we keep on using the abstraction with arbitrary objects being serialized to disk and being able to trace back and see if the model file (most commonly python pickle files) aren’t tampered with .

The ML field is doing work in that area: https://github.com/huggingface/safetensors

Re: Llamafile lets you distribute and run LLMs with a single file

#49
post #26

I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.

llamafile will run any compatible model you want. For example, if you download the LLaVA llamafile, you can still pass `-m wizardcoder.gguf` to override the default weights.

Re: Llamafile lets you distribute and run LLMs with a single file

#50
post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Thanks for the tip! Any chance this would run on a 2011 MacBook?
Post reply on HN