Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

231–240 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#231

Post by theptip »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

My pride as a technologist tells me I should be able to get any Python package up and running, but man, AI dependency management is a dumpster fire right now; adding GPU driver versions into the mix seems to make everything really brittle.

This seems like a great approach to compare multiple models, in particular.

Re: Llamafile lets you distribute and run LLMs with a single file

#232

Post by MacsHeadroom »

hnuser123456 wrote: post #221
sebmellen wrote: post #138

Earlier quoted context omitted.

Wow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web

Source on the newer GPT-4 model being worse at coding?

Everyone on twitter. Like 1/4th of my timeline for the past week has been people complaining that turbo won't complete code and instead returns things like "fill out the rest of the function yourself" or "consult a programming specialist for help on completing this section."

Re: Llamafile lets you distribute and run LLMs with a single file

#233

Post by pyinstallwoes »

dmd wrote: post #207
pyinstallwoes wrote: post #179

Earlier quoted context omitted.

What’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.

... is the javascript it's writing for you too 'woke' or something?

Lately it’s been refusing to write code at all. // implementation details here.

I think it’s the opposite of woke, it’s slept!

Re: Llamafile lets you distribute and run LLMs with a single file

#234

Post by twobitshifter »

hnuser123456 wrote: post #221
sebmellen wrote: post #138

Earlier quoted context omitted.

Wow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web

Source on the newer GPT-4 model being worse at coding?

It’s not that it’s worse, it’s just refusing to do coding without persistent prodding and the right prompts. Some think they are trying to do something with alignment, and maybe prevent it from giving code away so that they can upsell.

Re: Llamafile lets you distribute and run LLMs with a single file

#236

Post by raajg »

callmeed wrote: post #53
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

when I try to do this (MBP M1 Max, Sonoma) I get 'killed' immediately

Yep, same issue. and the error message is unhelpful

Re: Llamafile lets you distribute and run LLMs with a single file

#237

Post by NoOn3 »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Got this: Terminating on uncaught SIGILL.

Re: Llamafile lets you distribute and run LLMs with a single file

#238

Post by jart »

NoOn3 wrote: post #237
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Got this: Terminating on uncaught SIGILL.

I'm assuming you're on Apple Silicon? Please follow https://github.com/Mozilla-Ocho/llamafile/issues/11 which is tracking this. We've received multiple reports even though I personally haven't figured out how to reproduce it yet.

Re: Llamafile lets you distribute and run LLMs with a single file

#239

Post by [deleted] »

jart wrote: post #29
estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

You need at minimum a stock operating system install of: - Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer - MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64) - Windows 8+ (amd64) - FreeBSD 13+ (amd64, gpu should work in theory) - NetBSD 9.2+ (amd64, gpu should work in theory) - OpenBSD 7+ (amd64, no gpu support) - AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error an…

[deleted]

Re: Llamafile lets you distribute and run LLMs with a single file

#240

Post by czbond »

rudasn wrote: post #230
oefrha wrote: post #169

Earlier quoted context omitted.

Wrapping it in Docker makes it harder to access your filesystem without some dedicated jailbreak.

Volumes maybe? I don't see the issue here.

security - i have no idea if an executable will not be a malicious actor to my system. Dockerizing it, and removing access to system files - is the reason.

I am not trusting that a tradition virus / etc scanner will find stuff hidden in executable models.

Post reply on HN