llama.cpp

I think the best option right now, since Apple has raised prices and Mac minis are basically impossible to get your hands on, is to build your own micro-itx machine. I actually built a mini-itx machine, but it does restrict your options a bit.

The Arc series Intel GPUs are what I think make this possible. I built a machine with an Arc b50 - it runs Gemma 26b a4b qat at around 30tok/s with their MTP head and prompt processing sits at around 500 tok/s. The really beautiful thing about this setup is the entire energy envelope of this machine sits at 120w at full load - when idle, it's at 40w and i've done some work in ubuntu to basically intelligently hibernate, which drops it to 0 watts when not in use. You can use a raspberry pi and Wake on Lan to wake the machine up for a overall draw of around 5 watts when not in use.

All in all this machine cost me 1.4k to build - but if you used micro-itx instead of mini-itx parts you could do it for under 1k - it has just 16gb of ddr5 but you don't really need more if you use models that can fit in vram.

I think it's pretty incredible that you can run an actually useful coding agent on a machine with a power envelope that is less than an incandescent light bulb. If you go up to micro-itx you can do even large cards like an intel b60 with 24gb or a b70 with 32gb and run even more powerful models. For all of these intel GPU's you'll want to compile the latest llama.cpp version with SYCL support - they are getting speedups every day, so worth staying on the edge.

Then spend about 5-10 years minimum building web apps to get an idea how releasing a build and distribution works.

Very unimpressed by how everything in "AI" is brought to market.

It's like Apple Watch fanboys but with Linux style evangelism. Nobody wants that and it doesn't make us think you are "smart" and we're "dumb" if your tech is not easy to install and your docs are not easy to follow.

I use Ollama (CLI and their npm as a dev) + HF because it just works.

ps. ngram didn't work for me very well, but dedicated speculative model works very well

ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like:

    # over baseline at temp 0.6 (95 vs 45 tok/s), ~4x over naive layer-split baseline.
    Qwen3.6-27B-MTP-UD-Q8_K_XL:
      ./llama.cpp/llama-server \
        -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q8_K_XL \
        --no-mmproj \
        --parallel 1 \
        --kv-unified \
        --flash-attn on \
        --fit off \
        --split-mode tensor \
        -ngl 99 \
        --cache-type-k q8_0 \
        --cache-type-v q8_0 \
        --host 0.0.0.0 \
        --tools all \
        --jinja \
        --ctx-size 262144 \
        --spec-type draft-mtp \
        --spec-draft-n-max 6 \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --repeat-penalty 1.0 \
        --presence-penalty 1.1 \
        --threads 8 \
        --reasoning-budget 2048 \
        --reasoning on \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --reasoning-budget-message "reasoning budget consumed, time to answer now"

    ...

    Qwen: Qwen3.6
    Qwen3.6: Qwen3.6-27B
    Qwen3.6-35B-A3B: Qwen3.6-35B-A3B-MTP
    Qwen3.6-35B-A3B-MTP: Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL
    Qwen3.6-27B: Qwen3.6-27B-MTP
    Qwen3.6-27B-MTP: Qwen3.6-27B-MTP-UD-Q8_K_XL