411–420 of 652 posts
Earlier quoted context omitted.
> you're having issues handling files properly? I guess they were using ollama, which does not tell you where it puts the models it downloads.
Filelight / ncdu are my friends for finding random 30GB directories containing cached models.
Earlier quoted context omitted.
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
Qwen-3.6-35B-A3B was our "gateway drug" into switching our organisation to agent/harness-first coding. Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode. It continues to be a workhorse to this…
I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though.
DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models.
For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs.
But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.
In a 30M+ LOC repo we blew out Qwen3.6’s context thinking about a code review. This was via direct call to model. Do harnesses facilitate better context management, or is there something else to accommodate its smaller context window?
What was it reviewing? Was it just a "take a look at this pile of code over here"? Because I think asking any model to review 30M lines of code is a stretch...
This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days. And that makes me wonder if the trill…
They have 2 moats. The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are. The second is the…
Whilst these coding models improve, they seem to cater for multiple languages, and for many, a trimmed-down LLM that supports just one language would be fantastic. This would be smaller, more able to run on the hardware people have at home, realistically (even on the CPU). Having one LLM that knows assembler, java, bain, C etc is neat, but when you only work and use one language at a time, it would be kinda neat to h…
That's not how LLMs work. If you're talking about number of parameters, you wouldn't be able to reduce the size much by "removing" support for other languages.
In a 30M+ LOC repo we blew out Qwen3.6’s context thinking about a code review. This was via direct call to model. Do harnesses facilitate better context management, or is there something else to accommodate its smaller context window?
Earlier quoted context omitted.
Needs 0 investment and 0 committment? - You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harnes…
You might be anti-ai in the sense you aren’t comfortable with all your data being shipped back and forth to a third party.
Local AI is almost perfect. But its like all democracy: its history is marred with lots of crap.
Earlier quoted context omitted.
I didn't mention it above, but Laguna S is my other favorite model. I use Qwen a lot more, it's smaller and faster, but I like to switch to Laguna when I feel like I need a "heavy hitter" for certain huge or complex tasks.
What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
Earlier quoted context omitted.
> what level is this compared to other Western paid version? IMHO this is a difficult question to answer. Part of the power of paid models comes from the software supporting it. With local models, you have tons of workflows that can severely influence the quality of the result. In my personal experience, the SOTA models are way more consistent and can handle more complex questions. Part of that is (probably) because…
You absolutely need to let models access the Internet if you want consistently good results. Pretty much any non-trivial task requires the model to do things like look up APIs, code examples, or existing discussions of a given topic.
Reading the actual code is also always a better source of truth than docs anyway (this is true for people and LLMs). Just clone whatever libraries you use.
This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days. And that makes me wonder if the trill…
Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.