[flagged]
Jamesob's guide to running SOTA LLMs locally
191–193 of 193 posts
Re: Jamesob's guide to running SOTA LLMs locally
#192Earlier quoted context omitted.
I'm afraid prompts and clever arrangements of data don't really negate the parent post warnings. It's great if it works for you and your projects. Unfortunately, I can almost guarantee your approach will break down once you get a project large enough or switch to a less popular language. My favorite example is Godot; most local models just can't get it through their thick AI skull that code alone won't be enough to g…
There is no replacement for large models, indeed. And this is not the point I'm trying to make. There are numerous applications for self-hosted models. As a simplest example, when you ask "explain what this code does" advantage of large models is negligible. I tried Fable, "look at this repo, find all bugs" — yeah, neither Qwen nor Opus can do this. > I can almost guarantee your approach will break down once you get…
PHP is ancient in the current timeline of programming history. The Venn diagram of your code base and what it digested during training is basically a circle.
Re: Jamesob's guide to running SOTA LLMs locally
#193in my experience running models that have been heavily quantized(q4) or altered to some extent has never made me say “wow, this is so amazing”. On the contrary, the model ended up in the thrash bin after a few prompts. I have an RTX 6000 PRO with 96GB, and what I can run comfortably is Qwen 3.6 27B or MoE, Gemma 4 31B. This is as far as it goes when you run the model at full precision and maximum context length. They…
what you maybe forget here is the use case for people and businesses who can not send the data to 3rd party due to privacy/contractual reasons. This is what I'm looking at, we're bound by strict policies for data sharing outside of our premises.
yes, I understand the usecase. Where I'm coming from is quantized vs. unquantized. 4bit quants are lobotomizing the model heavily to the point that it's better to invest in some capable hardware than keep fighting the limitation. Refurbished server grade hardware is accessible. For the price of an RTX 6000 PRO you could probably get much more VRAM but 1-2 generations older.