>Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source. Heh, what a coincidence, just today one of my students presented research results which also confirmed this. He removed MLP from Qwen and the model still could do transformation tasks on input but lost knowledge.
> He removed MLP from Qwen and the model still could do transformation tasks on input but lost knowledge.
I'm so excited for this, nice work! Gemma4 edge models were promised to be great for agentic use, but have been really disappointing in all my tests. They fail at the most basic tool use scenarios. Have you run any tool-use benchmarks for Needle, or do you plan to? Would be great if you could add results to the repo if so.
Wait what? I've used DeepSeek V4 flash a lot and compsred to Gemma 4 E2B (ie. the smallest, event at q4), it consistently underperforms. In contrast to DS flash, I've found Gemma 4 to be incredibly precise and consistent with tool use.
>Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source. Heh, what a coincidence, just today one of my students presented research results which also confirmed this. He removed MLP from Qwen and the model still could do transformation tasks on input but lost knowledge.
Hmm.. this might make it feasible to build something like a command line program where you can optionally just specify the arguments in natural language. Although I know people will object to including an extra 14 MB and the computation for "parsing" and it could be pretty bad if everyone started doing that. But it's really interesting to me that that may be possible now. You can include a fine-tuned model that under…
I wonder if it will make sense to have a central natural language parser of sorts managed by the OS that includes a model like this? So all command line programs can hook into this model at runtime, and you can have adaptors for fine-tuning etc.
Yes I assume some kind of LLM system will eventually make its way into OS package registries and eventually there will be a standard of some sort. Already OpenAI's API is the de facto standard available from all providers. But who knows what port it's on. Maybe implement mDNS/DNS-SD or SSDP, and if not there it could just go through default ports like the Ollama one and check for a certain endpoint.