Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

71–80 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#71

Post by jart »

RecycledEle wrote: post #57

Fantastic. For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a huge number of dependencies? It seems like someone would have included Llama.cpp in their distro, ready-to-run. Yes, I'm an idiot.

llamafile runs on all Linux distros since ~2009. It doesn't have any dependencies. It'd probably even run as the init process too (if you assimilate it). The only thing it needs is the Linux 2.6.18+ kernel application binary interface. If you have an SELinux policy, then you may need to tune things, and on some distros you might have to install APE Loader for binfmt_misc, but that's about it. See the Gotchas in the README. Also goes without saying that llamafile runs on WIN32 too, if that's the world you're most comfortable with. It even runs on BSD distros and MacOS. All in a single file.

Re: Llamafile lets you distribute and run LLMs with a single file

#72

Post by throwaway743 »

jart wrote: post #43
xnx wrote: post #30

> Windows also has a maximum file size limit of 2GB for executables. You need to have llamafile and your weights be separate files on the Windows platform. The 4GB .exe ran fine on my Windows 10 64-bit system.

You're right. The limit is 4 gibibytes . Astonishingly enough, the llava-v1.5-7b-q4-server.llamafile is 0xfe1c0ed4 bytes in size, which is just 30MB shy of that limit. https://github.com/Mozilla-Ocho/llamafile/commit/81c6ad3251f...

Not at my windows machine to test this out right now, but wondering what you mean by having to store the weights in a separate file for wizardcoder, as a result of the 4gb executable limit. How does one go about this?

Thank you!

Re: Llamafile lets you distribute and run LLMs with a single file

#74

Post by phh »

lol768 wrote: post #62
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…

Last thing you need is to chmod +x the interpreter: chmod +x /usr/bin/ape (it is indeed not in the README)

Re: Llamafile lets you distribute and run LLMs with a single file

#75

Post by carbocation »

callmeed wrote: post #53
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

when I try to do this (MBP M1 Max, Sonoma) I get 'killed' immediately

Same on an M1 Max 64G, Ventura. Xcode is installed[1].

1 = ```

$ xcode-select --install

xcode-select: error: command line tools are already installed, use "Software Update" in System Settings to install updates

```

Re: Llamafile lets you distribute and run LLMs with a single file

#76

Post by gary_0 »

mercutio2 wrote: post #51
jart wrote: post #29

Earlier quoted context omitted.

You need at minimum a stock operating system install of: - Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer - MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64) - Windows 8+ (amd64) - FreeBSD 13+ (amd64, gpu should work in theory) - NetBSD 9.2+ (amd64, gpu should work in theory) - OpenBSD 7+ (amd64, no gpu support) - AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error an…

Apple Security will be excited to reach out to you to find out where you got a copy of macOS 15.6 :) I'm guessing this should be 13.6?

This is jart we are talking about. Perhaps, having made code Actually Portable in space, now she is doing time.

Re: Llamafile lets you distribute and run LLMs with a single file

#77

Post by russellbeattie »

dekhn wrote: post #26

I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.

I sorta see your point - it's kinda the equivalent of self-executable SQLite database pre-filled with data, or a Word document that contains the editor. There's lots of good reasons the data and apps are delivered separately.

That said, it does reduce the friction of getting an LLM up and running and the self-contained nature makes it sort of a dedicated program equivalent to Awk. It might open up possibilities, like the AI version of the Unix philosophy - do one thing well. A hyper optimized LLM could be used in pipeline of commands, for example.

Re: Llamafile lets you distribute and run LLMs with a single file

#78

Post by jart »

mercutio2 wrote: post #51
jart wrote: post #29

Earlier quoted context omitted.

You need at minimum a stock operating system install of: - Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer - MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64) - Windows 8+ (amd64) - FreeBSD 13+ (amd64, gpu should work in theory) - NetBSD 9.2+ (amd64, gpu should work in theory) - OpenBSD 7+ (amd64, no gpu support) - AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error an…

Apple Security will be excited to reach out to you to find out where you got a copy of macOS 15.6 :) I'm guessing this should be 13.6?

15.6 is a Darwin kernel version from 2018. It's the number `uname -a` reports. We should probably just switch to using XNU version numbers, which are in the 10000s now, so there's no confusion. I'm reasonably certain it works that far back, but I currently lack the ability to spin up old MacOS VMs for testing. Caveat emptor anyone not running MacOS on a recent version.

Re: Llamafile lets you distribute and run LLMs with a single file

#79

Post by mistrial9 »

great! worked easily on desktop Linux, first try. It appears to execute with zero network connection. I added a 1200x900 photo from a journalism project and asked "please describe this photo" .. in 4GB of RAM, it took between two and three minutes to execute with CPU-only support. The response was of mixed value. On the one hand, it described "several people appear in the distance" but no, it was brush and trees in the distance, no other people. There was a single figure of a woman walking with a phone in the foreground, which was correctly described by this model. The model did detect 'an atmosphere suggesting a natural disaster' and that is accurate.

thx to Mozilla and Justin Tunney for this very easy, local experiment today!

Post reply on HN