Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

51–60 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#51

Post by mercutio2 »

jart wrote: post #29
estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

You need at minimum a stock operating system install of: - Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer - MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64) - Windows 8+ (amd64) - FreeBSD 13+ (amd64, gpu should work in theory) - NetBSD 9.2+ (amd64, gpu should work in theory) - OpenBSD 7+ (amd64, no gpu support) - AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error an…

Apple Security will be excited to reach out to you to find out where you got a copy of macOS 15.6 :)

I'm guessing this should be 13.6?

Re: Llamafile lets you distribute and run LLMs with a single file

#52

Post by csdvrx »

Asmod4n wrote: post #25
rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

It’s just a zip file, updating it should be doable in place while it’s running on any non windows platform and you just need to swap that one file out you changed. When it’s running in server mode you could also possibly hot reload the executable without the user even having any downtime.

You could also change you code so that when it runs, it checks as early as possible if you have a file with a well known name (say ~/.freechat.run) and then switches to reading from it instead for the assets than can change.

You could have multiple updates my using say iso time and doing a sort (so that ~/.freechat.run.20231127120000 would be overriden by ~/.freechat.run.20231129160000 without making the user delete anything)

Re: Llamafile lets you distribute and run LLMs with a single file

#53

Post by callmeed »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

when I try to do this (MBP M1 Max, Sonoma) I get 'killed' immediately

Re: Llamafile lets you distribute and run LLMs with a single file

#54

Post by visarga »

zitterbewegung wrote: post #46

This is not to be dismissive but there is a security risk if we keep on using the abstraction with arbitrary objects being serialized to disk and being able to trace back and see if the model file (most commonly python pickle files) aren’t tampered with .

You just need to have a stray TXT file in your system, or even downloaded from internet that prompts the AI to hack your system. If your AI has Python sand box and that has vulnerabilities, you can be hacked by any web page or text file. And the AI would be able to study your computer and select the most juicy bits to send out. It would be like a sentient virus spread by simple text files (text bombs?).

Re: Llamafile lets you distribute and run LLMs with a single file

#55

Post by csdvrx »

dekhn wrote: post #42
simonw wrote: post #36

Earlier quoted context omitted.

Personally I really like the single file approach. If the weights are 4GB, and the binary code needed to actually execute them is 4.5MB, then the size of the executable part is a rounding error - I don't see any reason NOT to bundle that with the model.

I guess in every world I've worked in, deployment involved deploying a small executable which would run millions of times on thousands of servers, each instance loading a different model (or models) over its lifetime, and the weights are stored in a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device. The executable itself doesn't even contain the final model- just a descri…

> a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device

that assumption gets wrong very fast with nvme storage, even before you add herding effects

Re: Llamafile lets you distribute and run LLMs with a single file

#56

Post by abrinz »

I've been playing with various models in llama.cpp's GGUF format like this.

  git clone https://github.com/ggerganov/llama.cpp     

  cd llama.cpp

  make 

  # M2 Max - 16 GB RAM

  wget -P ./models https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF/resolve/main/openhermes-2.5-mistral-7b-16k.Q8_0.gguf
  
  ./server -m models/openhermes-2.5-mistral-7b-16k.Q8_0.gguf -c 16000 -ngl 32

  # M1 - 8 GB RAM 

  wget -P ./models https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF/resolve/main/openhermes-2.5-mistral-7b.Q4_K_M.gguf

  ./server -m models/openhermes-2.5-mistral-7b.Q4_K_M.gguf -c 2000 -ngl 32

Re: Llamafile lets you distribute and run LLMs with a single file

#57

Post by RecycledEle »

Fantastic.

For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a huge number of dependencies?

It seems like someone would have included Llama.cpp in their distro, ready-to-run.

Yes, I'm an idiot.

Re: Llamafile lets you distribute and run LLMs with a single file

#58

Post by simonw »

bilsbie wrote: post #50
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Thanks for the tip! Any chance this would run on a 2011 MacBook?

Justine says it needs MacOS 13.6+ - does that run on that machine?

Re: Llamafile lets you distribute and run LLMs with a single file

#59

Post by brucethemoose2 »

amelius wrote: post #21
tomwojcik wrote: post #15

Earlier quoted context omitted.

That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?

This is a bit like saying if you don't specify "--dram", the data will be stored on punchcards. From the user's point of view: they just want to run the thing, and as quickly as possible. If multiple programs want to use the GPU, then the OS and/or the driver should figure it out.

Llama.cpp allocates stuff to the GPU statically. It'd not really analogous to a game.

It should have a heuristic that looks at available VRAM by default, but it does not. Probably because this is vendor specific and harder than you would think, and they would rather not use external libraries.

Post reply on HN