Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to setup a local coding agent on macOS

ikyle.me

Post reply on HN
Print view View original

1–10 of 150 posts

Re: How to setup a local coding agent on macOS

#3

Post by c-hendricks »

Not sure you really need huggingface-cli to download anything if you're just using llama.cpp. You can pass `-hf ...` and it will download the models for you. Set `LLAMA_CACHE` to change where the downloads go:

  LLAMA_CACHE="models" ./llama-server \
    -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
    ...

Re: How to setup a local coding agent on macOS

#4

Post by dofm »

c-hendricks wrote: post #3

Not sure you really need huggingface-cli to download anything if you're just using llama.cpp. You can pass `-hf ...` and it will download the models for you. Set `LLAMA_CACHE` to change where the downloads go: LLAMA_CACHE="models" ./llama-server \ -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \ ...

Yes.

-hfd for the draft model.

Re: How to setup a local coding agent on macOS

#6

Post by dofm »

Useful stuff in here that I wish I'd seen a few days ago :-)

I am not convinced that the MTP setup for the QAT model adds very much in terms of speed on my M1 Max, but it is definitely worth experimenting with.

Fiddling about with local models has done so much for my conceptual understanding of what is going on.

FWIW and YMMV but I also found the Gemma 4 MTP head was occasionally breaking markup in Opencode, causing the thinking to display untidily and ultimately in some cases missing the stop token. So I've stopped using MTP there for now.

Recent Qwen 3.6 models have developer role support so it will occasionally surprise you with a structured multiple choice questionnaire.

Re: How to setup a local coding agent on macOS

#9

Post by c-hendricks »

dofm wrote: post #4
c-hendricks wrote: post #3

Not sure you really need huggingface-cli to download anything if you're just using llama.cpp. You can pass `-hf ...` and it will download the models for you. Set `LLAMA_CACHE` to change where the downloads go: LLAMA_CACHE="models" ./llama-server \ -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \ ...

Yes. -hfd for the draft model.

Nice, was wondering if there was a flag for the draft as well.

Not knocking huggingface-cli, just find it's much easier for people to try out this stuff when they can just

  mise use --global github:ggml-org/llama.cpp
  LLAMA_CACHE="models" llama-server \
    -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL \
    --host 0.0.0.0 \
    --port 11434 \
    ...
Post reply on HN