31–40 of 44 posts
Anybody was able to run this model in a server?
Yes, it runs in vllm happily. It gets >900TPS output reliably on a single 5090 with the nvfp4 model. It's clearly worse than vanilla 26B-A4B, and lacks some things like structured outputs, and gets some tool calls wrong. So you have to find a usecase or a hand rolled harness that leverages the cerebras-level TPS while not going off track during (even short) tasks.
If these models get good at coding it's going to force a rethink of how languages, compilers and test suite runners work. "AI changes everything" is a cliché by this point but I think it's actually true. If your model can reason and write code at 1500 toks/sec, then you should end up totally bottlenecked on CPU time the entire time a prompt is active. If you aren't, then you're losing wall time versus competitors. Bu…
no way llms can reason through (spring) java's stacktrace hell, and rust compilation is just too slow, i think golang is gonna be gold.
there's still JEPA to be integrated before AGI. Would DiffusionGemma be suitable candidate for DFlash 2?
Earlier quoted context omitted.
Yes, it runs in vllm happily. It gets >900TPS output reliably on a single 5090 with the nvfp4 model. It's clearly worse than vanilla 26B-A4B, and lacks some things like structured outputs, and gets some tool calls wrong. So you have to find a usecase or a hand rolled harness that leverages the cerebras-level TPS while not going off track during (even short) tasks.
And it fails on rocm of course. This engine is such a hassle on AMD.
RDNA4 is pretty similar to CDNA4, yet over a year after the release of "pro AI" cards like the r9700, they had basic kernels lacking in vllm (like w4a16 int4 kernels) while they were implemented in the datacenter CDNA4 cards.
AMD hardware runs well on llama.cpp because basically anything runs on llama.cpp, especially with vulkan. It's not high praise of AMD's software team to say llama.cpp runs well on their hardware
Right now running it locally is about 7-11 t/s for me (16GB 4080 and plenty of RAM overflow). If it doubled or possibly tripled this, thats a game-changer
If these models get good at coding it's going to force a rethink of how languages, compilers and test suite runners work. "AI changes everything" is a cliché by this point but I think it's actually true. If your model can reason and write code at 1500 toks/sec, then you should end up totally bottlenecked on CPU time the entire time a prompt is active. If you aren't, then you're losing wall time versus competitors. Bu…
no way llms can reason through (spring) java's stacktrace hell, and rust compilation is just too slow, i think golang is gonna be gold.
Appealing results... do we think there is scope to close the accuracy gap against AR models? or even leverage the "Bidirectional Reasoning and Self-Correction" into an overall advantage?
Performance-wise, I don't think diffusion decoding should be much worse than autoregressive decoding, see https://arxiv.org/html/2604.11035 .
As a local LLM user, I really want to see more small diffusion LLM/VLMs with decent performance coming.