71–80 of 132 posts
Earlier quoted context omitted.
LM Studio isn't free/libre/open source software, which misses the point of using open weights and open source LLMs in the first place.
Disagree, there are a lot of reasons to use open source local LLMs that aren't related to free/libre/oss principles. Privacy being a major one.
Why is ollama so many people’s go-to? Genuinely curious, I’ve tried it but it feels overly stripped down / dumbed down vs nearly everything else I’ve used. Lately I’ve been playing with Unsloth Studio and think that’s probably a much better “give it to a beginner” default.
Ollama user with the opposite question -- why not? What am I missing out on? I'm using it as the backend for playing with other frontend stuff and it seems to work just fine. And as someone running at 16gb card, I'm especially curious as to if I'm missing out on better performance?
Earlier quoted context omitted.
Even with the latest version of LM Studio and the latest runtimes I find that tool use fails 100% of the time with the following error: Error rendering prompt with jinja template: "Cannot apply filter "upper" to type: UndefinedValue". EDIT: The issue is addressed in LM Studio 0.4.9 (build 1), which auto-update wasn't picking up for me for some reason.
I googled it- supposed fixed template https://github.com/ggml-org/llama.cpp/issues/21347#issuecomm...
Which harness (IDE) works with this if any? Can I use it for local coding right now?
I've got a workaround for that called petsitter where it sits as a proxy between the harness and inference engine and emulates additional capabilities through clever prompt engineering and various algorithms.
They're abstractly called "tricks" and you can stack them as you please.
https://github.com/day50-dev/Petsitter
You can run the quantized model on ollama, put petsitter in front of it, put the agent harness in front of that and you're good to go
If you have trouble, file bugs. Please!
Thank you
edit: just checked, the ollama version supports everything
$ llcat -u http://localhost:11434 -m gemma4:latest --info
["completion", "vision", "audio", "tools", "thinking"]
Earlier quoted context omitted.
It's not just LLM sourced though, folks have literally tried this after the release with the 26A4B model and it wasn't very good. Maybe the dense ~31B model is worthwhile though.
Many Gemma implementations are or were broken on launch day. The first attempts to fix llama.cpp’s tokenizer were merged hours ago. Everyone hated Qwen3.5 at launch too because so many implementations were broken and couldn’t do tool calling. You need to ignore social media “I tried this and it sucks” echo chambers for new model releases.
If this is your first time using open weight models right after release, know that there are always bugs in the early implementations and even quantizations. Every project races to have support on launch day so they don’t lose users, but the output you get may not be correct. There are already several problems being discovered in tokenizer implementations and quantizations may have problems too if they use imatrix. S…
You seem like you know what you're talking about... what inference engine should I use? (linux, 4090) I keep having "I tried it but it sucks" issues mostly around tool calling and it's not clear if it's the model or ollama. And not one model in particular, any of them really.
Same thing happened when GPT-OSS launched, bunch of projects had "day-1" support, but in reality it just meant you could load the model basically, a bunch of them had broken tool calling, some chat prompt templates were broken and so on. Even llama.cpp which usually has the most recent support (in my experience) had this issue, and it wasn't until a week or two after llama.cpp that GPT-OSS could be fairly evaluated with it. Then Ollama/LM Studio updates their llama.cpp some days after that.
So it's a process thing, not "this software is better than that", and it heavily depends on the model.
Why is ollama so many people’s go-to? Genuinely curious, I’ve tried it but it feels overly stripped down / dumbed down vs nearly everything else I’ve used. Lately I’ve been playing with Unsloth Studio and think that’s probably a much better “give it to a beginner” default.