Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

221–230 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#221

Post by hnuser123456 »

sebmellen wrote: post #138
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Wow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web

Source on the newer GPT-4 model being worse at coding?

Re: Llamafile lets you distribute and run LLMs with a single file

#222

Post by whstl »

larodi wrote: post #188
rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

Perhaps another unpopular opinion that can get the comment outright down-voted, but still... While jart's work is very interesting in nature and execution, commendable stuff indeed of a person with very high IQ and discipline, I still wonder whether Justine simply can't get over the fact they got kicked out of the llama.cpp project (yes, I understand jart is frequenting HN, and also let's agree llama.cpp is at least…

I read the Github repository README and the comments here and I found absolutely nothing that could suggest the need for the first two paragraphs you wrote. It seems this stems from a misconception from your side about the purpose of this project.

About your question in the third paragraph: This is totally orthogonal to GGUF, and a cursory reading of the README shows that it does uses GGUF. This is not about a new universal LLM format, this is about packing it in a universal executable that runs everywhere, using Cosmopolitan.

Some examples do pack the executable and GGUF weights together in a single file, but that's not dissimilar from an self-executing zip, the only difference is that this executable is not OS-specific, so you can use the same exact binary for macOS or Linux, for example.

Re: Llamafile lets you distribute and run LLMs with a single file

#223

Post by whstl »

joenot443 wrote: post #210
pdntspa wrote: post #90

Earlier quoted context omitted.

I don't get this obsession with 0-click everything. It is really annoying when you don't want to install everything to your main hard drive. I have all my models downloaded, organized, and ready-to-go but apps won't even ask for that, instead it presumes I am an idiot and downloads it (again!) for me. At least Makeayo asks where my models are now. It's obnoxious that I have to use symlinks for comfy/automatic.... All…

The “obsession” isn’t for developers like you or I, it’s for non-technical but curious non-engineers who would still like to play around with an LLM. I think you’re imposing your own very specific (and solvable!) gripe onto an unrelated project and it makes for pretty clumsy commentary.

Yep.

But also: It might not be for a developer like you, but it is for a developer like me.

I enjoy writing software, but I don't particularly enjoy futzing with building things outside my day-to-day work, and on systems I don't write myself. If it was up to me everything would be one click.

Things like this are like accessibility: it benefits me even though I don't particularly need it.

Re: Llamafile lets you distribute and run LLMs with a single file

#224

Post by whstl »

pdntspa wrote: post #181
mft_ wrote: post #149

Earlier quoted context omitted.

If I'm understanding (and agreeing with) your gripe correctly, isn't it two solutions to the same perceived problem? My experience is that the world of Python dependency management is a mess which sometimes works, and sometimes forces you to spend hours-to-days searching for obscure error messages and trying maybe-fixes posted in Github issues for some other package, just in case it helps. This sometimes extends furt…

I get that, my issue is when the model is coupled with the app, or the app just presumes I don't have it downloaded and doesn't ask me otherwise. This is like basic configuration stuff... What I suspect is happening is that people are cargo-culting zero-click installations. It seems rather fashionable right now.

In the README of the project (the TFA of this whole thread) there is the option to download the app without the model:

"You can also also download just the llamafile software (without any weights included) from our releases page, or directly in your terminal or command prompt"

There is no cargo-culting going on. Some of us do legitimately appreciate it.

Re: Llamafile lets you distribute and run LLMs with a single file

#225

Post by CaptainFever »

epaga wrote: post #189

Hmmm no matter what image I upload, it's somehow not seeing it. Got a chuckle out of this, though: --- User: Describe the image in the most detailed way possible Llama: The image is a close-up of an empty, dark room with no visible objects or people. It appears to be a black background photo that highlights the darkness and emptiness of the space. This could potentially represent an artistic representation or an actu…

That's comedic. I love it.

Re: Llamafile lets you distribute and run LLMs with a single file

#226

Post by ElectricalUnion »

pdntspa wrote: post #181
mft_ wrote: post #149

Earlier quoted context omitted.

If I'm understanding (and agreeing with) your gripe correctly, isn't it two solutions to the same perceived problem? My experience is that the world of Python dependency management is a mess which sometimes works, and sometimes forces you to spend hours-to-days searching for obscure error messages and trying maybe-fixes posted in Github issues for some other package, just in case it helps. This sometimes extends furt…

I get that, my issue is when the model is coupled with the app, or the app just presumes I don't have it downloaded and doesn't ask me otherwise. This is like basic configuration stuff... What I suspect is happening is that people are cargo-culting zero-click installations. It seems rather fashionable right now.

Is this the sentiment around?

Is having everything normalized in your system that worth it? I would say having (some) duplicates in your system is mostly fine, better that having some spooky-action-at-a-distance break things when you don't expect.

I expect the future is something like Windows's WinSxS, NixOS's /nix/store, pnpm's .pnpm-store where that deduping isn't "online" but it still is somewhat automated and hidden from you.

Re: Llamafile lets you distribute and run LLMs with a single file

#227

Post by mark_l_watson »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Simon, does this also work well on Apple Silicon?

EDIT: never mind, other commenters here answered my question: yes it does work. I have a M2Pro with 32 G on chip memory - amazingly effective for experimenting with open LLMs.

Re: Llamafile lets you distribute and run LLMs with a single file

#228

Post by ukuina »

belter wrote: post #167
reisse wrote: post #165

Earlier quoted context omitted.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

Deploy your models into a Kubernetes cluster and let them fight for resources to death? A modern Roman Gladiators circus with Models?

Give them write access to your control-plane and the first one to write syntactically correct IaC wins!

Re: Llamafile lets you distribute and run LLMs with a single file

#229

Post by ElectricalUnion »

larodi wrote: post #188
rgbrgb wrote: post #8

Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…

Perhaps another unpopular opinion that can get the comment outright down-voted, but still... While jart's work is very interesting in nature and execution, commendable stuff indeed of a person with very high IQ and discipline, I still wonder whether Justine simply can't get over the fact they got kicked out of the llama.cpp project (yes, I understand jart is frequenting HN, and also let's agree llama.cpp is at least…

> llama.cpp already compiles and runs pretty much everywhere.

Well, it simplifies things when you don't need to compile things.

Also, you literally can't download or compile the wrong binary by mistake, it's the same binary for all supported processor/OSes Cartesian product matrix.

> Why introduce one more container?

It makes stuff more convenient.

`application/zip` is also a ubiquitous standard. I doubt anyone is being "introduced to it".

I also appreciate the fact that tooling for handling `application/zip` is very widespread, so you don't need totally bespoke tooling to retrieve the models from inside a `llamafile`.

> Who benefits from binary distribution of this sort?

Anyone that doesn't have a compiler SDK on their computer.

Re: Llamafile lets you distribute and run LLMs with a single file

#230

Post by rudasn »

oefrha wrote: post #169
reisse wrote: post #165

Earlier quoted context omitted.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

Wrapping it in Docker makes it harder to access your filesystem without some dedicated jailbreak.

Volumes maybe? I don't see the issue here.
Post reply on HN