Which is a smaller model, that gives good output and that works best with this. I am looking to run this on lower end systems.
I wonder if someone has already tried https://github.com/jzhang38/TinyLlama, could save me some time :)
Llamafile lets you distribute and run LLMs with a single file
201–210 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#202Earlier quoted context omitted.
do you... have any plans to upgrade? A gen 2011 computer is going to get harder and harder to make work. even a used macbook from like 2019 would probably be a steal at this point, and that's 8 years further along All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available
makes one think that perhaps Apple isn't a good long term choice...
Apple is a fine long-term choice (speaking as a recent linux advocate actually, lol). But that computer is 12 years old
Re: Llamafile lets you distribute and run LLMs with a single file
#203I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Anyone have any tuning tips? I messed with some of the configs and now it's mostly hallucinating answers or going off the rails
Re: Llamafile lets you distribute and run LLMs with a single file
#204I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
I just tried asking it a question: > User: What is the third planet from the sun? > Llama: The third planet from the sun is called Mars.
Llama is just from the future. That is all…
Re: Llamafile lets you distribute and run LLMs with a single file
#205I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
when I try to do this (MBP M1 Max, Sonoma) I get 'killed' immediately
Same, process gets killed immediately for me.
Re: Llamafile lets you distribute and run LLMs with a single file
#206Earlier quoted context omitted.
You need at minimum a stock operating system install of: - Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer - MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64) - Windows 8+ (amd64) - FreeBSD 13+ (amd64, gpu should work in theory) - NetBSD 9.2+ (amd64, gpu should work in theory) - OpenBSD 7+ (amd64, no gpu support) - AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error an…
Time to go amd, poor old me, Intel MB Air 2018 (zsh: exec format error, Darwin Kernel Version 22.2.0, MacOS Ventura 13.1).
You need to upgrade to zsh 5.9+ or run `sh -c ./llamafile`. See the Gotchas section of the README.
Re: Llamafile lets you distribute and run LLMs with a single file
#207Earlier quoted context omitted.
The best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.
What’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.
... is the javascript it's writing for you too 'woke' or something?
Re: Llamafile lets you distribute and run LLMs with a single file
#208Earlier quoted context omitted.
The best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.
What’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.
In my experience, the deepseek-coder-instruct family is at least as good as gpt-3.5-turbo on python. Even the 1.3b models are very good (and run okay on cpu), although you should use larger if you have the vram. There are even larger models (30b+) if you are drowning in vram, but I don't think they perform much better at coding than deepseek-coder-instruct 6.7b.
3-4gb vram or cpu (1.3b): https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct...
Alternative for chat (1.3b): https://huggingface.co/TheBloke/evolvedSeeker_1_3-GGUF
Alternative for chat (3b): https://huggingface.co/TheBloke/open-llama-3b-v2-wizard-evol...
6-8gb vram (6.7b): https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...
Re: Llamafile lets you distribute and run LLMs with a single file
#209I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
This can truly revolutionalize education and access, feels like what I hoped the One Laptop Per Child project would do, this could actually achieve. We just need someone with a heap of funds to package this up into a very inexpensive machine and distribute them.
Re: Llamafile lets you distribute and run LLMs with a single file
#210Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…
I don't get this obsession with 0-click everything. It is really annoying when you don't want to install everything to your main hard drive. I have all my models downloaded, organized, and ready-to-go but apps won't even ask for that, instead it presumes I am an idiot and downloads it (again!) for me. At least Makeayo asks where my models are now. It's obnoxious that I have to use symlinks for comfy/automatic.... All…
The “obsession” isn’t for developers like you or I, it’s for non-technical but curious non-engineers who would still like to play around with an LLM.
I think you’re imposing your own very specific (and solvable!) gripe onto an unrelated project and it makes for pretty clumsy commentary.