51–60 of 85 posts
"Multi-token prediction gives a free speedup of up to 2x on many models" - at the expense of halving prompt processing speed
MTP has no effect on prompt processing.
I strongly considered it, but the one thing that scares me away from wanting to do the spark is you basically have to use nvidia's linux (from what I've read) and it doesn't appear the nvidia is interested in upstreaming their kernel changes. I'm avoiding where possible buying electronics where support is controlled by the manufacturer and not me.
If I understand your comment correctly, I think he addresses that in the first few paragraphs: > The DGX Spark runs “DGX OS” but it is in fact just plain old Ubuntu 24.04 with some additions. If you want, you can just install another Linux distribution easily (Fedora works well), although there may be a couple of weird bugs with the Realtek Ethernet driver so the NVIDIA version of the Linux kernel has a couple of pat…
I thought I'd read that the GPU needed extra kernel patches to properly work. If it's just the Ethernet driver that seems a lot more appealing.
I thought it was device tree as well, so great that it's actual ACPI.
Earlier quoted context omitted.
AMD is barely trying. The 395+ Max Strix Halo was launched January 2025. gfx1151 was not listed in the ROCm compatibility matrix for ROCm 7.2.4 [0]. This is the previous version of ROCm. It's only finally received support in ROCm 7.14.0 [1]! It literally just started receiving support last week. [0] https://rocm.docs.amd.com/en/docs-7.2.4/compatibility/compat... [1] https://rocm.docs.amd.com/en/docs-7.14.0/compatibil…
Is it to me or the iGPU of Ryzen 9950X is not supported by ROCm?
"Multi-token prediction gives a free speedup of up to 2x on many models" - at the expense of halving prompt processing speed
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 4
Earlier quoted context omitted.
MTP has no effect on prompt processing.
Interesting - in my setup (llama.cpp rtx5090 qwen-3.6 27b) prompt processing with mtp is almost half vs non mtp. Sounds like I need to investigate what is wrong.
edit: I recommend building recent llama.cpp from source, I've been updating about once a week, as there has been a fair amount of work related to MTP recently. If you're running a lot of tool calling on Qwen you might also benefit from one of the bugfixed chat templates like the Froggeric version.
Unfortunitly, as others have noted, the DGX OS experience... sucks. My hope is that the RTX Spark (which looks to be the exact same stack, sans the high capacity network interface) will help this get a bit more attention, but nVidia's long long long war with the open source community is not helping. Focusing on mainlining kernel support would go a long way to getting the community to be supportive.
Of course, a massive regression just hit Linux 7+/7.1 plus for ROCm hosts, so it's just rough everywhere.
Please don’t buy a DGX Spark unless all three of these are true: - You value simplicity more than performance or price-to-performance. - You accept that the hardware will depreciate rapidly. - You’re prepared to buy two or four of them. OR: - You want to run frontier models right now as cheaply as possible - You want to run high-parameter models on a 15a breaker/line Otherwise, get a normal, high-bandwidth GPU. A sin…
I'm actually confused by the post - indeed why choosing the 27B dense and the 35B MoE models? Given the structure of the post I honestly have the impression that the Spark was purchased out of curiosity more than for a specific purpose (LLMs).
I did not intend to imply that the post author was advocating that they're great for inference, as they're obviously not.
Please don’t buy a DGX Spark unless all three of these are true: - You value simplicity more than performance or price-to-performance. - You accept that the hardware will depreciate rapidly. - You’re prepared to buy two or four of them. OR: - You want to run frontier models right now as cheaply as possible - You want to run high-parameter models on a 15a breaker/line Otherwise, get a normal, high-bandwidth GPU. A sin…
I'm actually confused by the post - indeed why choosing the 27B dense and the 35B MoE models? Given the structure of the post I honestly have the impression that the Spark was purchased out of curiosity more than for a specific purpose (LLMs).
Especially at larger quanitizations (Q4 is pretty crap).