191–200 of 302 posts
Noob question - how might I call this from my Python script? Say as a replacement gpt3.5 turbo of sorts. Is there an option without GUI? This is great thank you, very user friendly (exhibit a: me)
Earlier quoted context omitted.
Simon Willison's blog post has a screenshot. It's worth a read. https://simonwillison.net/2023/Nov/29/llamafile/
Thank you, is there a way to select a different model? How does the model perform? Just general question if anyone else knows the answers while I try and clear space on my laptop ( why these things fill up so fast!)
You can use this to run any if the thousands of of GGUF models on Hugging Face, see note here: https://simonwillison.net/2023/Nov/29/llamafile/#llamafile-t...
Extremely cool and Justine Tunney / jart does incredible portability work [0], but I'm kind of struggling with the use-cases for this one. I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it.…
Perhaps another unpopular opinion that can get the comment outright down-voted, but still... While jart's work is very interesting in nature and execution, commendable stuff indeed of a person with very high IQ and discipline, I still wonder whether Justine simply can't get over the fact they got kicked out of the llama.cpp project (yes, I understand jart is frequenting HN, and also let's agree llama.cpp is at least…
Mass adoption comes from ease-of-use, which includes ease-of-deployment.
Most of the HN crowd is savvy enough to run HF models directly in their terminal, but there's a reason Ollama, LM Studio, and Faraday are so popular.
Until LLM runners are part of the base OS, methods like this are extremely helpful to reduce friction.
I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…
This is amazing. How does result quality compare to GPT4 for image analysis?
Earlier quoted context omitted.
I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.
I see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume…
There are tradeoffs hers. For some people the tradeoff of a single executable file with everything in it compared to setting up a Docker system is worthwhile.
Earlier quoted context omitted.
I see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume…
If you want to use Docker then you can go ahead - I don't see anyone claiming that this brand new, highly experimental project should be used by everyone instead of Docker. There are tradeoffs hers. For some people the tradeoff of a single executable file with everything in it compared to setting up a Docker system is worthwhile.
Why does it feel like everyday I see some new example of stupidity on HN.
Please explain. This feels like a worthwhile effort to push LLMs towards mass-adoption.
Currently which are the minimum system requirements for running these models?
You need at minimum a stock operating system install of: - Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer - MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64) - Windows 8+ (amd64) - FreeBSD 13+ (amd64, gpu should work in theory) - NetBSD 9.2+ (amd64, gpu should work in theory) - OpenBSD 7+ (amd64, no gpu support) - AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error an…
Earlier quoted context omitted.
>Extremely cool and Justine Tunney / jart does incredible portability work [0], [x] Doubt. That user was caught stealing code and banned from llama.cpp by its creator (your [2] citation) https://news.ycombinator.com/item?id=35411909 Maybe the same thing is happening here. Plagiarism of code.
What are you on about? There was no stealing and there was no plagiarism. They made a PR that was built on top of another PR. The authorship information was preserved in the git history, and there was no attempt at deception. They also supposedly collaborated with the author of the original PR (which was never denied by either of them). All of this is totally normal working practice. Those allegations of "stealing" j…
See this post https://news.ycombinator.com/item?id=35418066