Viewing profile — ModelForge

Good question, it's 100%. I.e., the developers usually share a reference implementation with e.g., Hugging Face transformers to load their weights, and from there on you can read t…

And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"

No the compiled version is actually faster. From that table, the A100 tok/sec (larger is faster) numbers are: - Eager: 28 - Compiled: 128 And - KV cache eager: 26 - KV cache compil…

Could be an artifact of the small size not fully taking advantage of the GPU. For example, for the slightly larger Qwen3 0.6B model the A100 is faster (you can see it when scrollin…

I'd say the common ones (besides educational) are - private, on-device models (possibly with lower latency than models via web API); also edge devices - algorithm research (faster …

I think GPT-4.5 was potentially the original GPT-5 model that was larger and pre-trained on more data. Too bad it was too expensive to deploy at scale so that we never saw the RL-e…

The ollama one uses even less (around 13 GB), which is nice. Apparently the gpt-oss team also shared the mxfp4 optimizations for metal

Good point. LLMs lower the barrier to entry if someone has enough resources because those architectures are more robust to tweaks given one throws enough compute and data at them. …

I’ve been using the ollama version (uses about 13 Gb RAM on macOS) and haven’t had that issue yet. I wonder if that’s maybe an issue of the llama.cpp port?