771–780 of 809 posts
Earlier quoted context omitted.
> no need to learn, just ask it to do it for you And that's how skills die.
And why is this skill important, if a machine can do it ? What's the last time you ploughed your field with oxen ?
Earlier quoted context omitted.
Available models aren’t really trending upward in size. Not like I thought they would, anyway. They’re trending to be the right size to be good. Qwen3.6-35B is not as good as Qwen3.6-27B. The larger model is faster, but a lot dumber; it gets caught in loops, makes crazy mistakes, and is just not as good. It’s bigger, but it is nowhere near as good as the 27B variant.
Qwen3.6-35B-A3B is worse than 27B because it's an MoE and 27B is dense. 35B only passes each token through 3B of its total parameters, whereas 27B sends each token through all 27B parameters.
Earlier quoted context omitted.
The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. I don't know how much serious hands-free agentic coding I will ever do on my MacBook alone, but I do know that I would not have got so far into understanding this without tinkering with local models, llama.cpp, LM Studio, and LM Studio and all th…
> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Cloud models are (much) faster, they don't consume so much power/generate heat, they have much bigger (LLM) context, they're much more precise and they have a much wider (engineering) context of the given problem. Except privacy and use cases that are blocked by cloud models (e.g. reverse engineering), local LLMs are cur…
So if you're just putzing around, sure, cloud will be fine, but if you're building a workflow/business around these things, you're essentially making yourself an indentured servant.
Earlier quoted context omitted.
[flagged]
Yikes, you broke the site guidelines badly with this post. Could you please review https://news.ycombinator.com/newsguidelines.html and stick to them? You're welcome to make your substantive points thoughtfully, just not aggressively.
I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…
I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…
Earlier quoted context omitted.
Can you say a bit more about this? The bad tool calling has made me give up on using Gemma for my Hermes and a personal recipe site. I have only downloaded from Ollama.
Ollama is not recommended [0], use llama.cpp or more specifically Unsloth Studio which wraps llama.cpp and which has an API mode you can use to hook into Hermes or another agent. Unsloth make both the Studio and the quants which fix various issues with many models [1] as well as implementing new features like MTP and QAT support much sooner than other teams. In general you should read r/LocalLLaMa as it has a lot of…
1. vLLM
2. sglang
3. (nvidia only) TRT-LLM
4. llama.cpp (mac only, the above are better for non-mac)
If you're not using one of the above, you're doing it wrong
Earlier quoted context omitted.
I'm currently fiddling with a DGX Spark and Qwen3.6-35B-A3B (specifically Qwen3.6-35B-A3B-NVFP4 under vLLM, with EAGLE3 speculative decoding via eagle3-dogacel-vllm), and it's pretty okay in terms of smarts. The speed is relatively usable at about 50 tok/sec with a 256k context window, and it's definitely smart enough to one-shot some basic coding tasks. I had it doing reverse engineering/disassembly of some ancient…
Compared to a dynamic quant like Unsloth's UD-Q4_K_XL, which keeps some important parameters in higher precision, a basic NVFP4 quant seems to do a lot more damage to the model unless it is carefully calibrated. I would recommend using llama-server if you're just on a single Spark. You get access to dynamic quants like that more easily, the performance is not that different from vLLM most of the time these days, and…
Earlier quoted context omitted.
I'm currently fiddling with a DGX Spark and Qwen3.6-35B-A3B (specifically Qwen3.6-35B-A3B-NVFP4 under vLLM, with EAGLE3 speculative decoding via eagle3-dogacel-vllm), and it's pretty okay in terms of smarts. The speed is relatively usable at about 50 tok/sec with a 256k context window, and it's definitely smart enough to one-shot some basic coding tasks. I had it doing reverse engineering/disassembly of some ancient…
Looping is a common problem with the Qwen models. I've had good luck using --repeat-penalty=1.1 with llama.cpp and 27B. vLLM should have a similar option.
Earlier quoted context omitted.
Compared to a dynamic quant like Unsloth's UD-Q4_K_XL, which keeps some important parameters in higher precision, a basic NVFP4 quant seems to do a lot more damage to the model unless it is carefully calibrated. I would recommend using llama-server if you're just on a single Spark. You get access to dynamic quants like that more easily, the performance is not that different from vLLM most of the time these days, and…
[flagged]
I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts.
I’m glad you’ve had a better experience? I can only speak to the experiences that I have had repeatedly. For at least a month, people on the official Spark forum were claiming you just couldn’t run MiMo-V2.5 on a single Spark, because they refused to use anything other than vLLM, while I was doing it just fine on llama-server with 200k+ of context.
And llama-server is “worse” in what specific ways? I was specific with my comment. The usual complaint was the lack of MTP/Eagle3 support in llama-server, but that is solved now. Now the main difference is a minor hit to prompt processing speed, at most, if you’re using a single Spark.
Too many people on the Spark forum are closed minded to the idea that vLLM is not the solution to every problem.
llama-server also comes with a truly excellent built-in web chat interface these days, which includes the ability to connect to MCPs so the models can be used agentically through a conversational interface even from my phone. What does vLLM offer? Yeah… nothing. And options like Open WebUI seem really bloated.
For a cluster of multiple Sparks, the pain of vLLM is still worthwhile, as I already said before. Or if you’re running some kind of major production workload, I guess? Instead of a single user, few agent setup like most people.