Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My first impressions on ROCm and Strix Halo

blog.marcoinacio.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: My first impressions on ROCm and Strix Halo

#31

Post by ZiiS »

sidkshatriya wrote: post #26

> It seems that things wouldn't work without a BIOS update: PyTorch was unable to find the GPU. This was easily done on the BIOS settings: it was able to connect to my Wifi network and download it automatically. Call me traditional but I find it a bit scary for my BIOS to be connecting to WiFi and doing the downloading. Makes me wonder if the new BIOS blob would be secure i.e. did the BIOS connect over securely over…

I have never seen a BIOS that didn't allow offline updates? However SSL is much less processing then a WPA2 WiFi stack, I would certainly expect this to be fully secure and boycot a manufacturer who failed. Conversely updating your BIOS without worrying if your OS is rooted is nice.

Re: My first impressions on ROCm and Strix Halo

#32

Post by rdslw »

anko wrote: post #9

I would be interested to know what speeds you can get from gemma4 26b + 31b from this machine. also how rocm compares to triton.

## performance data for token generation using lmstudio

- gemma4-31b normal q8 -> 5.1 tok/s

- gemma4-31b normal q16 -> 3.7 t/s

- gemma4-31b distil q16 -> 3.6 t/s

- gemma4-31b distil q8 -> 5.7 tok/s (!)

- gemma4-26b-a4b ud q8kxl -> 38 t/s (!)

- gemma4-26b-a4b ud q16 -> 12 t/s

- gemma4-26b-a4b cl q8 -> 42 t/s (!)

- gemma4-26b-a4b cl q16 -> 12 t/s

- qwen3.5-35b-a3b-UD@q6_k -> 52 t/s (!)

- qwen3.5-35b-a3b-uncensored-hauhaucs-aggressive@q8_0 -> 34 tok/s (!)

- qwen3.5-35b-a3b-uncensored-hauhaucs-aggressive@bf16 -> 11 tok/s

- qwen3.5-27b-claude-4.6-opus-reasoning-distilled-v2 q8 -> 8 tok/s

- qwen3.5 122B A10B MXFP4 Mo qwen3.5-122b-a10b (q4) -> 11 tok/s

- qwen3.5-122b-a10b-uncensored-hauhaucs-aggressive (q6) -> 10 tok/s

Re: My first impressions on ROCm and Strix Halo

#33

Post by moffkalast »

suprjami wrote: post #27

If you are using quants below Q8 then get them from Unsloth or Bartowski. They are higher quality than the quants you can make yourself due to their imatrix datasets and selective quantisation of different parts of the model. For Qwen 3.5 Unsloth did 9 terabytes of quants to benchmark the effects of this: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

That used to be a good suggestion, and it still most likely is if you're using a recent Nvidia dGPU, but absolutely not for iGPUs like the Halo/Point or Arc LPG. The problem is bf16.

In short, even lower quants leave some layers at original precision and llama.cpp in its endless wisdom does not do any conversion when loading weights and seeing what your card supports, so every time you run inference it gets so surprised and hits a brick wall when there's no bf16 acceleration. Then it has to convert to fp16 on the fly or something else which can literally drop tg by half or even more. I've seen fp16 models literally run faster than Q8 on Arc despite being twice the size with the same bandwidth and it's expectedly similar [0] on AMD.

Models used to be released as fp16 which was fine, then Gemma did native bf16 and Bartowski initially came up with a compatibility thing where they converted bf16 to fp32 then fp16 and used that for quants. Most models are released as bf16 these days though and Bartowski's given up on doing that (while Unsloth never did that to begin with). So if you do want max speed, you kinda have to do static quants yourself and follow the same multi-step process to remove all the stupid bf16 weights from the model. I don't get why this can't be done once at model load ffs, but this is what we've got.

[0] https://old.reddit.com/r/LocalLLaMA/comments/1r0b7p8/free_st...

Re: My first impressions on ROCm and Strix Halo

#34

Post by trvz »

sidkshatriya wrote: post #30
imp0cat wrote: post #28

Earlier quoted context omitted.

Isn't this pretty much standard in this day and age? HP for example also has this option in BIOS for their laptops (but you still can either download the BIOS blob manually in Linux or use the automatic updater in Windows if you want).

> Isn't this pretty much standard in this day and age? If something is "standard" nowadays does it mean it is the right way to go ? One of my main issues is that this means your BIOS has to have a WiFi software stack in it, have a TLS stack in it etc. Basically millions of lines of extra code. Most of it in a blob never to be seen by more than a few engineers. Though in another a way allowing BIOS to perform self upd…

I fully expect any BIOS to have millions of unnecessary lines of code already though. May as well have a bit more for user convenience.

Re: My first impressions on ROCm and Strix Halo

#35

Post by adrian_b »

fluoridation wrote: post #23
fwipsy wrote: post #21

Earlier quoted context omitted.

Quad-channel RAM is common on consumer desktops. Strix Halo has *8* channels, and also very fast RAM (soldered RAM can be faster than dimms because the traces are shorter.)

Quad channel memory is not common on consumer desktops, it's a strictly HEDT and above feature. The vast majority of consumer desktops have 2 channels or fewer.

One should no longer use the word "channel" because the width of a channel differs between various kinds of memories, even among those that can be used with the same CPU (e.g. between DDR and LPDDR or between DDR4 and DDR5).

For instance, now the majority of desktops with DDR5 have 4 channels, not 2 channels, but the channels are narrower, so the width of the memory interface is the same as before.

To avoid ambiguities, one should always write the width of the memory interface.

Most desktop computers and laptop computers have 128-bit memory interfaces.

The cheapest desktop computers and laptop computers, e.g. those with Intel Alder Lake N/Twin Lake CPUs, and also many smartphones and Arm-based SBCs, have 64-bit memory interfaces.

Cheaper smartphones and Arm-based SBCs have 32-bit memory interfaces.

Strix Halo and many older workstations and many cheaper servers have 256-bit memory interfaces.

High-end servers and workstations have 768-bit or 512-bit memory interfaces.

It is expected that future high-end servers will have 1024-bit memory interfaces per socket.

GPUs with private memory have usually memory interfaces between 192-bit and 1024-bit, but newer consumer GPUs have usually narrower memory interfaces than older consumer GPUs, to reduce cost. The narrower memory interface is compensated by faster memories, so the available bandwidth in consumer GPUs has been increased much slower than the increase in GDDR memory speed would have allowed.

Re: My first impressions on ROCm and Strix Halo

#36

Post by adrian_b »

moffkalast wrote: post #33
suprjami wrote: post #27

If you are using quants below Q8 then get them from Unsloth or Bartowski. They are higher quality than the quants you can make yourself due to their imatrix datasets and selective quantisation of different parts of the model. For Qwen 3.5 Unsloth did 9 terabytes of quants to benchmark the effects of this: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

That used to be a good suggestion, and it still most likely is if you're using a recent Nvidia dGPU, but absolutely not for iGPUs like the Halo/Point or Arc LPG. The problem is bf16. In short, even lower quants leave some layers at original precision and llama.cpp in its endless wisdom does not do any conversion when loading weights and seeing what your card supports, so every time you run inference it gets so surpri…

The CPU of Strix Halo has good BF16 acceleration, like any other Zen 4/Zen 5 CPU (the future Zen 6 will add FP16 acceleration).

I do not know about its GPU, which might have only FP16.

So it is likely that the right inference strategy would be to run any BF16 computations on the Strix Halo CPU, while running the quantized computations on its GPU.

Re: My first impressions on ROCm and Strix Halo

#37

Post by adrian_b »

ZiiS wrote: post #31
sidkshatriya wrote: post #26

> It seems that things wouldn't work without a BIOS update: PyTorch was unable to find the GPU. This was easily done on the BIOS settings: it was able to connect to my Wifi network and download it automatically. Call me traditional but I find it a bit scary for my BIOS to be connecting to WiFi and doing the downloading. Makes me wonder if the new BIOS blob would be secure i.e. did the BIOS connect over securely over…

I have never seen a BIOS that didn't allow offline updates? However SSL is much less processing then a WPA2 WiFi stack, I would certainly expect this to be fully secure and boycot a manufacturer who failed. Conversely updating your BIOS without worrying if your OS is rooted is nice.

Updating your BIOS without worrying if your OS is rooted can be easily and more securely done from an USB memory.

The BIOSes recent enough to be able to connect through the Internet normally have the option to use a USB memory from inside the BIOS setup.

Some motherboards can update the BIOS from a USB memory even without a CPU in the socket.

Re: My first impressions on ROCm and Strix Halo

#38

Post by tssge »

adrian_b wrote: post #36
moffkalast wrote: post #33

Earlier quoted context omitted.

That used to be a good suggestion, and it still most likely is if you're using a recent Nvidia dGPU, but absolutely not for iGPUs like the Halo/Point or Arc LPG. The problem is bf16. In short, even lower quants leave some layers at original precision and llama.cpp in its endless wisdom does not do any conversion when loading weights and seeing what your card supports, so every time you run inference it gets so surpri…

The CPU of Strix Halo has good BF16 acceleration, like any other Zen 4/Zen 5 CPU (the future Zen 6 will add FP16 acceleration). I do not know about its GPU, which might have only FP16. So it is likely that the right inference strategy would be to run any BF16 computations on the Strix Halo CPU, while running the quantized computations on its GPU.

The GPU has INT4, INT8, BF16 and FP16. Notably no FP8 or FP4.The official GPTQ-Int4 release from Qwen is a great quant for this but custom kernels are still rare for this hardware.

Re: My first impressions on ROCm and Strix Halo

#39

Post by bityard »

sidkshatriya wrote: post #26

> It seems that things wouldn't work without a BIOS update: PyTorch was unable to find the GPU. This was easily done on the BIOS settings: it was able to connect to my Wifi network and download it automatically. Call me traditional but I find it a bit scary for my BIOS to be connecting to WiFi and doing the downloading. Makes me wonder if the new BIOS blob would be secure i.e. did the BIOS connect over securely over…

You don't HAVE to update the bios over wifi, fwupd is perfectly able to do it as well.

Re: My first impressions on ROCm and Strix Halo

#40

Post by bityard »

If you just want to run models, most of TFA is taking the scenic route.

All you really need is podman, toolbx, and the Strix Halo toolbox images from https://github.com/kyuz0/amd-strix-halo-toolboxes. Then you just download your ggufs and hand them to llama-server.

Yes, there are other solutions that are a bit more hand-holdy, but if you already know how to use docker/podman and just want to get something working in an evening, this works too.

Post reply on HN