Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

171–180 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#171
post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

Very nice; works perfect on Ubuntu 20.04. Doing 8 tokens/s on a pretty crappy server.

Perfectly on Fedora 39 on old ( and I mean old...) machines. This is actually shocking...shocking good...

Re: Llamafile lets you distribute and run LLMs with a single file

#172
post #165

Earlier quoted context omitted.

Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

Homelabbers like me have a docker swarm cluster / k8s cluster so this def helps!

Re: Llamafile lets you distribute and run LLMs with a single file

#173

Earlier quoted context omitted.

woah, this is fast. On my M1 this feels about as fast as GPT-4.

How good is it in comparison

The best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.

Re: Llamafile lets you distribute and run LLMs with a single file

#174
It’s intriguing to observe how this model is hallucinating. I presented it with Albert Bierstadt’s painting “In the Mountains” (https://www.wikiart.org/en/albert-bierstadt/in-the-mountains...). To my surprise, it maintains that cows are scattered throughout the scene, despite being told otherwise.

User:

Llama: The image features a serene landscape with mountains in the background and a river flowing through it. A picturesque scene unfolds as two trees are visible near the water, adding to the natural beauty of the environment. In addition to these elements, there is an interesting detail: several cows can be seen standing on or around the grassy area next to the body of water. The presence of these animals creates a sense of tranquility and harmony with nature in this idyllic setting.

User: There are no cows. Could it be that you've mistaken rocks for cows?

Llama: No, the rocks in this image are not mistaken for cows. They can be seen near a lake with mountains in the background and trees surrounding it. The painting captures a serene landscape where nature is beautifully displayed. This picturesque scene showcases both the majestic mountain range and the tranquil body of water, creating an idyllic atmosphere that evokes feelings of peacefulness and relaxation.

User: There are no cows!

Llama: The image depicts a serene scene of mountains and water. A beautiful lake is surrounded by majestic snow-capped mountains, creating an idyllic landscape. In the foreground, there are trees adding to the natural beauty of the environment.

A few birds can be seen flying in the sky above the lake, further enhancing the peaceful atmosphere. The presence of a boat on the water adds another element of leisure and enjoyment to this picturesque scene. Overall, it is an inviting image that showcases nature's wonders with its combination of mountains, lake, trees, birds, and boats.

Re: Llamafile lets you distribute and run LLMs with a single file

#175
post #165

Earlier quoted context omitted.

Popped it into a docker setup: https://github.com/tluyben/llamafile-docker to save even more keystrokes.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

Personally I wanted to see if this llama can generate the required docker files to run itself just from pasting in the instructions from the my parent commenter and asking for docker/docker-compose. It did and it works.

Re: Llamafile lets you distribute and run LLMs with a single file

#176
post #50

Earlier quoted context omitted.

Thanks for the tip! Any chance this would run on a 2011 MacBook?

do you... have any plans to upgrade? A gen 2011 computer is going to get harder and harder to make work. even a used macbook from like 2019 would probably be a steal at this point, and that's 8 years further along All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available

makes one think that perhaps Apple isn't a good long term choice...

Re: Llamafile lets you distribute and run LLMs with a single file

#177

Fantastic. For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a huge number of dependencies? It seems like someone would have included Llama.cpp in their distro, ready-to-run. Yes, I'm an idiot.

On a fresh install of Ubuntu 23.10, I made it executable (`chmod +x llamafile`) and it worked just like that. After installing `nvidia-cuda-toolkit` it also worked with my GPU. So far the most seamless method I've tried.

Re: Llamafile lets you distribute and run LLMs with a single file

#178
post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

I just tried asking it a question:

> User: What is the third planet from the sun?

> Llama: The third planet from the sun is called Mars.

Re: Llamafile lets you distribute and run LLMs with a single file

#179

Earlier quoted context omitted.

How good is it in comparison

The best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.

What’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.

Re: Llamafile lets you distribute and run LLMs with a single file

#180
post #151

Earlier quoted context omitted.

Way cheaper? I thought that 1K Tokens (in+out) cost 0.04 USD in GPT-4 Turbo, which is roughly one larger chat response (2 screens). To reach parity with ChatGPT Plus pricing you need thus to use less than 500 such responses per month via API. For GPT-4 the pricing is double that (0.09 USD per 1K). So only 200 larger interactions to reach 20 USD cost. Or am I wrong?

It depends on your usage; for me the plus sub is much cheaper than if I use the api directly, but I use it a lot for everything I do.

[flagged]
Post reply on HN