Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My first impressions on ROCm and Strix Halo

blog.marcoinacio.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: My first impressions on ROCm and Strix Halo

#41

Post by anaisbetts »

SwellJoe wrote: post #11
roenxi wrote: post #8

I thought the point of something like Strix Halo was to avoid ROCm all together? AMDs strategy seems to have been to unify GPU/CPU memory then let people write their own libraries. The industry looks like it's started to move towards Vulkan. If AMD cards have figured out how to reliably run compute shaders without locking up (never a given in my experience, but that was some time ago) then there shouldn't be a reason…

You misunderstand the point, and ROCm. The GPU and CPU share memory, that doesn't mean you don't need to interact with the GPU, anymore. You can use Vulkan instead of ROCm on Radeon GPUs, including on the Strix Halo (and for a while, Vulkan was more likely to work on the Strix Halo, as ROCm support was slow to arrive and stabilize), but you need something that talks to the GPU. Current ROCm, 7.2.1, works quite well o…

This hasn't been my experience, ROCm is usually not only a bit slower for me (~32 t/s vs ~43 t/s on the main model I use), it is way less reliable; any upgrade in kernel version or AMD driver and suddenly everything is broken

Re: My first impressions on ROCm and Strix Halo

#42

Post by rpdillon »

spoaceman7777 wrote: post #10

I'm somewhat confused as to why this is on the front page. It doesn't go into any real detail, and the advice it gives is... not good. You should definitely not be quantizing your own gguf's using an old method like that hf script. There are lots of ways to run LLMs via podman (some even officially recommended by the project!). The chip has been out for almost a year now, and its most notable (and relevant-to-AI) fea…

Agreed. Been running a Strix Halo box since mid-2025. Lemonade builds of llama.cpp with Unsloth or Bartowski quants have proven to be excellent.

Re: My first impressions on ROCm and Strix Halo

#43

Post by data-ottawa »

seemaze wrote: post #18

Check out the officially supported project Lemonade[0] by AMD. It has gfx1151 specific builds of vLLM, llama.cpp, comfy-ui, and even a PR to merge a Strix Halo port of Apple’s MLX[1] with a quick and easy install. [0] https://www.amd.com/en/developer/resources/technical-article... [1] https://github.com/lemonade-sdk/lemonade/issues/1642

I don’t think lemonade includes a comfyui wrapper, it does have stable diffusion support built in though.

Re: My first impressions on ROCm and Strix Halo

#44

Post by data-ottawa »

Linux kernel 7 enables the NPU on Linux. You can use fastflowLM with lemonade now.

It is quite slow, but if you want to compute embeddings in the background it’s fine.

I didn’t find it more energy efficient than just using the GPU for time insensitive tasks though.

Re: My first impressions on ROCm and Strix Halo

#45

Post by fwipsy »

phonon wrote: post #29
fwipsy wrote: post #21

Earlier quoted context omitted.

Quad-channel RAM is common on consumer desktops. Strix Halo has *8* channels, and also very fast RAM (soldered RAM can be faster than dimms because the traces are shorter.)

4 DIMMS =/= 4 channels

I knew that, but I still thought most desktops with 4 dimm slots supported quad-channel memory. I guess I was wrong.

Re: My first impressions on ROCm and Strix Halo

#46

Post by stebalien »

moffkalast wrote: post #33
suprjami wrote: post #27

If you are using quants below Q8 then get them from Unsloth or Bartowski. They are higher quality than the quants you can make yourself due to their imatrix datasets and selective quantisation of different parts of the model. For Qwen 3.5 Unsloth did 9 terabytes of quants to benchmark the effects of this: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

That used to be a good suggestion, and it still most likely is if you're using a recent Nvidia dGPU, but absolutely not for iGPUs like the Halo/Point or Arc LPG. The problem is bf16. In short, even lower quants leave some layers at original precision and llama.cpp in its endless wisdom does not do any conversion when loading weights and seeing what your card supports, so every time you run inference it gets so surpri…

At least for qwen3.5, it looks like unsloth has updated their quantization algorithms to avoid bf16. See the march 5th update:

https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/discussi...

I assume they're applying the same technique going forward, but I have no idea how to determine if this is the case.

Re: My first impressions on ROCm and Strix Halo

#47

Post by moffkalast »

tssge wrote: post #38
adrian_b wrote: post #36

Earlier quoted context omitted.

The CPU of Strix Halo has good BF16 acceleration, like any other Zen 4/Zen 5 CPU (the future Zen 6 will add FP16 acceleration). I do not know about its GPU, which might have only FP16. So it is likely that the right inference strategy would be to run any BF16 computations on the Strix Halo CPU, while running the quantized computations on its GPU.

The GPU has INT4, INT8, BF16 and FP16. Notably no FP8 or FP4.The official GPTQ-Int4 release from Qwen is a great quant for this but custom kernels are still rare for this hardware.

Must be a case of the hardware being there and the software not actually supporting it then.

Re: My first impressions on ROCm and Strix Halo

#48

Post by SwellJoe »

anaisbetts wrote: post #41
SwellJoe wrote: post #11

Earlier quoted context omitted.

You misunderstand the point, and ROCm. The GPU and CPU share memory, that doesn't mean you don't need to interact with the GPU, anymore. You can use Vulkan instead of ROCm on Radeon GPUs, including on the Strix Halo (and for a while, Vulkan was more likely to work on the Strix Halo, as ROCm support was slow to arrive and stabilize), but you need something that talks to the GPU. Current ROCm, 7.2.1, works quite well o…

This hasn't been my experience, ROCm is usually not only a bit slower for me (~32 t/s vs ~43 t/s on the main model I use), it is way less reliable; any upgrade in kernel version or AMD driver and suddenly everything is broken

It can be tricky to get/keep ROCm working, but around 7.2 it became reliable and as fast as or faster than ROCm 6.4.

And, I think the first response time of ROCm is pretty consistently faster than Vulkan, even if Vulkan has a slightly higher token rate. Though I don't see that big of a different on token rates, either. Honestly, though, I haven't done enough real testing to know for sure. The benchmarks Donato Capitella posts (https://kyuz0.github.io/amd-strix-halo-toolboxes/) have been my guide on what to run in what way, and the performance of most things that can run on the Strix Halo are Fast Enough(tm) such that I don't agonize about performance. When Vulkan was all that worked with llama.cpp, that's what I used. Now that ROCm is reliable, I'm using ROCm. ROCm feels faster, maybe just because it processes prompts faster and starts typing the answer fast (at a rate faster than I can read it, so when it starts answering is the more important metric even if faster token rate would lead to it finishing faster).

In short: If ever I'm doing something that will take many hours to complete, and I need to optimize it, I'll do some tests first to be sure I'm using the optimal path. Otherwise, as long as ROCm is working, I'll probably just keep using it.

Re: My first impressions on ROCm and Strix Halo

#49

Post by ethbr1 »

sidkshatriya wrote: post #30
imp0cat wrote: post #28

Earlier quoted context omitted.

Isn't this pretty much standard in this day and age? HP for example also has this option in BIOS for their laptops (but you still can either download the BIOS blob manually in Linux or use the automatic updater in Windows if you want).

> Isn't this pretty much standard in this day and age? If something is "standard" nowadays does it mean it is the right way to go ? One of my main issues is that this means your BIOS has to have a WiFi software stack in it, have a TLS stack in it etc. Basically millions of lines of extra code. Most of it in a blob never to be seen by more than a few engineers. Though in another a way allowing BIOS to perform self upd…

> If something is "standard" nowadays does it mean it is the right way to go ?

Next thing you'll be telling me that you have a problem piping internet hosted install scripts directly into shell!

Re: My first impressions on ROCm and Strix Halo

#50

Post by fluoridation »

adrian_b wrote: post #35
fluoridation wrote: post #23

Earlier quoted context omitted.

Quad channel memory is not common on consumer desktops, it's a strictly HEDT and above feature. The vast majority of consumer desktops have 2 channels or fewer.

One should no longer use the word "channel" because the width of a channel differs between various kinds of memories, even among those that can be used with the same CPU (e.g. between DDR and LPDDR or between DDR4 and DDR5). For instance, now the majority of desktops with DDR5 have 4 channels, not 2 channels, but the channels are narrower, so the width of the memory interface is the same as before. To avoid ambiguiti…

>now the majority of desktops with DDR5 have 4 channels, not 2 channels

Source? I just looked up two random X870E boards from Gigabyte and both are dual channel.

>To avoid ambiguities, one should always write the width of the memory interface.

They're incomparable quantities. More channels support more parallel operations, while a wider bus at a constant frequency supports higher throughput.

The bus width is not even that useful of a metric. It's more useful to talk about bits per second, which is the product of bus width and frequency.

Post reply on HN