Wondering if anyone has played around with the Strix Halo based systems with 128GB of unified memory for local AI, and what their experience has been so far in running different models:
- Practical limit on number of parameters x quantization
- What you've been using it for
- Which models and inference engines combine well
Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
1–7 of 7 posts
Re: Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
#2Own a framework desktop 128gb.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
Re: Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
#3Own a framework desktop 128gb. GPU bandwidth limits the amount of tokens/sec that you get. I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens. Haven't tested the new laguna S 2.1 yet. Another thing I liked was that I had enough ram to also run an embedding open model…
Thanks for the detailed response. My goal is to deploy small to medium-sized LLMs on it for things that I'd rather keep private and local, i.e. automation around smart home, but also tracking finances and portfolio, aggregating & summarizing news feeds and blog posts and so on. For these purposes, medium intelligence at low TPS seems good enough to me.
But a large part of it is also wanting to learn more about serving these models, and potentially some small-scale experiments with training and fine tuning.
Re: Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
#4I'm running qwen3.6, gpt-oss and embeddinggemma with Ollama on Ryzen 7 8700G + 96 GB DDR5 with 12-14 tokens/sec. Consider this as a floor. On Strix Halo it will run 4-6 times faster depending on memory bandwidth. For my local task 14 tok/s is quite enough for the price I paid.
Re: Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
#5I'm running qwen3.6, gpt-oss and embeddinggemma with Ollama on Ryzen 7 8700G + 96 GB DDR5 with 12-14 tokens/sec. Consider this as a floor. On Strix Halo it will run 4-6 times faster depending on memory bandwidth. For my local task 14 tok/s is quite enough for the price I paid.
So no GPU? Just a strong processor? I am considering building a 'in my house' system. Timing of this post is very fortuitous.
Re: Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
#6I'm running qwen3.6, gpt-oss and embeddinggemma with Ollama on Ryzen 7 8700G + 96 GB DDR5 with 12-14 tokens/sec. Consider this as a floor. On Strix Halo it will run 4-6 times faster depending on memory bandwidth. For my local task 14 tok/s is quite enough for the price I paid.
So no GPU? Just a strong processor? I am considering building a 'in my house' system. Timing of this post is very fortuitous.
Ryzen 7 8700G is an APU, and it has built-in Radeon 780M GPU (12 CUs, ~12.6 TFLOPS, plus an NPU). This is not very strong, but this is floor which runs local models for me and saves a lot of paid tokens. Potentially this setup may be upgraded to AI MAX when AM5 APUs will become available in box packages (now they are OEM-availabe only). The most expensive part of my system is DDR5, and it may be reused in case of upgrade. Actually, this is my self-hosted NAS (TrueNAS Scale) which also runs Ollama and serves as my local inference machine.
Re: Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
#7With 128 GB of unified memory, large models can fit into memory, but memory bandwidth quickly becomes the limiting factor. With long contexts in particular, the KV cache can consume a lot of memory and bandwidth. Quantizing the KV cache to Q8 roughly halves its memory usage, making it one of the most effective optimizations. MoE models are also well suited to this hardware because only a portion of their parameters is active for each token. As a result, a 122B-A10B model can run better than a dense 70B model.