Any DGX Spark users in this thread? What's your favourite model to run on it?
Without doubt, dsv4-flash-0731. Original weights; needs two connected DGX.
Thanks, I however have just one DGX.
Why your local LLM feels dumber than it is
201–210 of 233 posts
Re: Why your local LLM feels dumber than it is
#202Re: Why your local LLM feels dumber than it is
#203I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.
I’ve been using the mlx version with orb studio an opencode
Re: Why your local LLM feels dumber than it is
#204[flagged]
Re: Why your local LLM feels dumber than it is
#205Earlier quoted context omitted.
You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other opensource code. You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context. You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code. We do th…
> We do the finetuning only on small semantic data were it helps a lot. This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data? > There are companies though which ahve this exact problem with programming languages you normally don't see. This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantiall…
So we have free text paragraphs which describe machines. Our model detects if the machine in question is of the main category we are looking for, then our finetuned model extracts from the free form text semantic information about the machine.
Like color, features, horsepower etc.
This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while.
We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time.
----
If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above.
Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common.
If this is not an option at all:
You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples.
The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand.
You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you.
Like you give an LLM all the relevant context for it, then you let it generate pairs:
{ "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" }
(I have no clue about APL this is just a random example I asked an LLM to generate).
You might have luck and finding communities with the same issue you face.
Re: Why your local LLM feels dumber than it is
#206Earlier quoted context omitted.
Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.
If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s. It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`,…
Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun.
That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding.
I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking.
EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+
Re: Why your local LLM feels dumber than it is
#207I have been using open source LLMs locally for the past 5 months like Qwen, Llama, DeepSeek and others, and I have also noticed that current local models feel significantly dumber than closed source commercial models like ChatGPT, Claude, and Gemini. One main reason I think is the amount, variety, and quality of original authentic data on which they are being trained on, and also the training method plays a significa…
They are also an order of magnitude difference in number of parameters and that matters a lot.
Re: Why your local LLM feels dumber than it is
#208"I can't wait to run this new sota model locally. I'll just use the quantized version that is certain to be better than [other model I'm running]." This is fast becoming one of my top old-man-yells-at-clouds pet peeves. Reported performance metrics are ONLY good for the exact model weights . Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs. Quantizing a good model do…
Especially when there are many ways to do the quantization. You can get quite different results from different methods.
Re: Why your local LLM feels dumber than it is
#209> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…
Thank you for your responses. They've been very helpful.
I'm going to test the llama.cpp binaries directly. That seems to me to be the closest to what Ollama is trying to achieve, after all.
Re: Why your local LLM feels dumber than it is
#210Earlier quoted context omitted.
Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).…
There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/c…
Right, well I'm walking the hounds right now but I think I got something like this
https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md
SO, if that isn't official it explains the results I got. They were dreadful.