I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
Very nice; works perfect on Ubuntu 20.04. Doing 8 tokens/s on a pretty crappy server.
Llamafile lets you distribute and run LLMs with a single file
171–180 of 302 posts
Re: Llamafile lets you distribute and run LLMs with a single file
#172Earlier quoted context omitted.
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
Re: Llamafile lets you distribute and run LLMs with a single file
#173Earlier quoted context omitted.
woah, this is fast. On my M1 this feels about as fast as GPT-4.
How good is it in comparison
Re: Llamafile lets you distribute and run LLMs with a single file
#174User:
Llama: The image features a serene landscape with mountains in the background and a river flowing through it. A picturesque scene unfolds as two trees are visible near the water, adding to the natural beauty of the environment. In addition to these elements, there is an interesting detail: several cows can be seen standing on or around the grassy area next to the body of water. The presence of these animals creates a sense of tranquility and harmony with nature in this idyllic setting.
User: There are no cows. Could it be that you've mistaken rocks for cows?
Llama: No, the rocks in this image are not mistaken for cows. They can be seen near a lake with mountains in the background and trees surrounding it. The painting captures a serene landscape where nature is beautifully displayed. This picturesque scene showcases both the majestic mountain range and the tranquil body of water, creating an idyllic atmosphere that evokes feelings of peacefulness and relaxation.
User: There are no cows!
Llama: The image depicts a serene scene of mountains and water. A beautiful lake is surrounded by majestic snow-capped mountains, creating an idyllic landscape. In the foreground, there are trees adding to the natural beauty of the environment.
A few birds can be seen flying in the sky above the lake, further enhancing the peaceful atmosphere. The presence of a boat on the water adds another element of leisure and enjoyment to this picturesque scene. Overall, it is an inviting image that showcases nature's wonders with its combination of mountains, lake, trees, birds, and boats.
Re: Llamafile lets you distribute and run LLMs with a single file
#175Earlier quoted context omitted.
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
Re: Llamafile lets you distribute and run LLMs with a single file
#176Earlier quoted context omitted.
Thanks for the tip! Any chance this would run on a 2011 MacBook?
do you... have any plans to upgrade? A gen 2011 computer is going to get harder and harder to make work. even a used macbook from like 2019 would probably be a steal at this point, and that's 8 years further along All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available
Re: Llamafile lets you distribute and run LLMs with a single file
#177Fantastic. For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a huge number of dependencies? It seems like someone would have included Llama.cpp in their distro, ready-to-run. Yes, I'm an idiot.
Re: Llamafile lets you distribute and run LLMs with a single file
#178I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
> User: What is the third planet from the sun?
> Llama: The third planet from the sun is called Mars.
Re: Llamafile lets you distribute and run LLMs with a single file
#179Earlier quoted context omitted.
How good is it in comparison
The best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.
Re: Llamafile lets you distribute and run LLMs with a single file
#180Earlier quoted context omitted.
Way cheaper? I thought that 1K Tokens (in+out) cost 0.04 USD in GPT-4 Turbo, which is roughly one larger chat response (2 screens). To reach parity with ChatGPT Plus pricing you need thus to use less than 500 such responses per month via API. For GPT-4 the pricing is double that (0.09 USD per 1K). So only 200 larger interactions to reach 20 USD cost. Or am I wrong?
It depends on your usage; for me the plus sub is much cheaper than if I use the api directly, but I use it a lot for everything I do.