Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

21–30 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#21

Post by amelius »

tomwojcik wrote: post #15
amelius wrote: post #12

> you pass the --n-gpu-layers 35 flag (or whatever value is appropriate) to enable GPU This is a bit like specifying how large your strings will be to a C program. That was maybe accepted in the old days, but not anymore really.

That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?

This is a bit like saying if you don't specify "--dram", the data will be stored on punchcards.

From the user's point of view: they just want to run the thing, and as quickly as possible. If multiple programs want to use the GPU, then the OS and/or the driver should figure it out.

Re: Llamafile lets you distribute and run LLMs with a single file

#22

Post by Hedepig »

estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

I am currently tinkering with this all, you can download a 3b parameter model and run it on your phone. Of course it isn't that great, but I had a 3b param model[1] on my potato computer (a mid ryzen cpu with onboard graphics) that does surprisingly well on benchmarks and my experience has been pretty good with it.

Of course, more interesting things happen when you get to 32b and the 70b param models, which will require high end chips like 3090s.

[1] https://huggingface.co/TheBloke/rocket-3B-GGUF

Re: Llamafile lets you distribute and run LLMs with a single file

#23

Post by rgbrgb »

estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

In my experience, if you're on a mac it's about the file size * 150% of RAM to get it working well. I had a user report running my llama.cpp app on a 2017 iMac with 8GB at ~5 tokens/second. Not sure about other platforms.

Re: Llamafile lets you distribute and run LLMs with a single file

#25

Post by Asmod4n »

rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

It’s just a zip file, updating it should be doable in place while it’s running on any non windows platform and you just need to swap that one file out you changed. When it’s running in server mode you could also possibly hot reload the executable without the user even having any downtime.

Re: Llamafile lets you distribute and run LLMs with a single file

#27

Post by omeze »

dekhn wrote: post #26

I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.

Eh, this is exploring a more “static link” approach for local use and development vs the more common “dynamic link” that API providers offer. (Imperfect analogy since this is literally like a DLL but… whatever). Probably makes sense for private local apps like a PDF chatter.

Re: Llamafile lets you distribute and run LLMs with a single file

#28

Post by tbalsam »

Asmod4n wrote: post #25
rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

It’s just a zip file, updating it should be doable in place while it’s running on any non windows platform and you just need to swap that one file out you changed. When it’s running in server mode you could also possibly hot reload the executable without the user even having any downtime.

> in place

._.

Pain.

Re: Llamafile lets you distribute and run LLMs with a single file

#29

Post by jart »

estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

You need at minimum a stock operating system install of:

- Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer

- MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64)

- Windows 8+ (amd64)

- FreeBSD 13+ (amd64, gpu should work in theory)

- NetBSD 9.2+ (amd64, gpu should work in theory)

- OpenBSD 7+ (amd64, no gpu support)

- AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error and refuse to run. This means, if you have an Intel CPU, it needs to be Intel Core or newer (circa 2006+), and if you have an AMD CPU, then it needs to be Bulldozer or newer (circa 2011+). If you have a newer CPU with AVX or better yet AVX2, then llamafile will utilize your chipset features to go faster. No support for AVX512+ runtime dispatching yet.

- ARM64 microprocessors must have ARMv8a+. This means everything from Apple Silicon to 64-bit Raspberry Pis will work, provided your weights fit into memory.

I've also tested GPU works on Google Cloud Platform and Nvidia Jetson, which has a somewhat different environment. Apple Metal is obviously supported too, and is basically a sure thing so long as xcode is installed.

Post reply on HN