201–210 of 235 posts
Earlier quoted context omitted.
Llamafile: https://github.com/Mozilla-Ocho/llamafile
It's actually more difficult to use on linux (compared to ollama) because of the weird binfmt contortions you have to go through.
Earlier quoted context omitted.
Llamafile is great but solves a slightly different problem very well: how do I easily download and run a single model without having any infrastructure in place first? Ollama solves the problem of how I run many models without having to deal with many instances of infrastructure.
You don't need any infrastructure for llamafiles, you just download and run them (everywhere).
The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.
You'll find that Ollama is also distributed under an MIT license[2]. It's fine to disagree with their priorities and lack of transparency. But trying to argue how they use code from other repositories that permit such a thing is tilting at windmills, IMHO.
[0] https://github.com/ggerganov/llama.cpp/blob/master/LICENSE
Earlier quoted context omitted.
Can you please explain why you think they may be operating in bad faith?
Not parent, but same feeling. First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library. Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture. Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "M…
Granted, they could be a lot more helpful in providing information on how you do this. But this feature exists, at least.
Earlier quoted context omitted.
You’re coming at it from a point of knowledge. Read the first sentence of the Ollama website against the first paragraph of kobold’s GitHub. Newcomers don’t have a clue what “running a GGUF model..” means. It’s written by tech folk without an understanding of the audience.
Ollama is also written for technical/developer users, by accident (it seems), even though they don't want it to be strictly for technical users. I've opened a issue asking them to make it more clear that Ollama is for technical users, but they seem confident people with no terminal experience can and will also use Ollama: https://github.com/ollama/ollama/issues/7116
Earlier quoted context omitted.
I’m not sure I understand what this comment is responding to. Wouldn’t a distilled Deepseek still use the same tokenizer? I’m not claiming they are using llama in their backend. I’m just saying they are likely using a lower-parameter model too.
The small models that have been published as part of the DeepSeek release are not a "distilled DeepSeek", they're fine-tuned varieties of Llama and Qwen. DeepSeek may have smaller models internally that are not Llama- or Qwen-based but if so they haven't released them.
So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…
Earlier quoted context omitted.
For Intel, OpenVINO should be the preferred route. I dont follow AMD, but Vulkan is just the common denominator here.
If you support Vulkan, you support almost every GPU out there in the consumer market across all hardware vendors. It's an amazing fallback option. I agree they should also support OpenVINO, but compared to Vulkan OpenVINO is a tiny market.
If you run your local llm in the least performant way possible on tour overly expensive GPU, then you are not making value of your purchase.
Vulkan is a fallback option is all.
I even see people running on their CPU because some apps dont support their hardware and llama.cpp made it even possible. It is still a really bad idea.
Its just goes to show there’s still much to do.
This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?
I used both. I had a terrible time with llama, and did not realise it until I used ollama. I owned an RTX2070, and followed the llama instructions to make sure it was compiling with GPU enabled. I then hand-tweaked settings (numgpulayers) to try to make it offload as much as possible to the GPU. I verified that it was using a good chunk of my GPU ram (via nvidia-smi), and confirmed that with-gpu was faster than cpu-o…