Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My first impressions on ROCm and Strix Halo

blog.marcoinacio.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: My first impressions on ROCm and Strix Halo

#21

Post by fwipsy »

spoaceman7777 wrote: post #10

I'm somewhat confused as to why this is on the front page. It doesn't go into any real detail, and the advice it gives is... not good. You should definitely not be quantizing your own gguf's using an old method like that hf script. There are lots of ways to run LLMs via podman (some even officially recommended by the project!). The chip has been out for almost a year now, and its most notable (and relevant-to-AI) fea…

Quad-channel RAM is common on consumer desktops. Strix Halo has *8* channels, and also very fast RAM (soldered RAM can be faster than dimms because the traces are shorter.)

Re: My first impressions on ROCm and Strix Halo

#23

Post by fluoridation »

fwipsy wrote: post #21
spoaceman7777 wrote: post #10

I'm somewhat confused as to why this is on the front page. It doesn't go into any real detail, and the advice it gives is... not good. You should definitely not be quantizing your own gguf's using an old method like that hf script. There are lots of ways to run LLMs via podman (some even officially recommended by the project!). The chip has been out for almost a year now, and its most notable (and relevant-to-AI) fea…

Quad-channel RAM is common on consumer desktops. Strix Halo has *8* channels, and also very fast RAM (soldered RAM can be faster than dimms because the traces are shorter.)

Quad channel memory is not common on consumer desktops, it's a strictly HEDT and above feature. The vast majority of consumer desktops have 2 channels or fewer.

Re: My first impressions on ROCm and Strix Halo

#24

Post by SwellJoe »

sabedevops wrote: post #16
SwellJoe wrote: post #14

Earlier quoted context omitted.

> CUDA solved that problem. CUDA is a proprietary Nvidia product. CUDA solved the problem for Nvidia chips. On AMD GPUs, you use ROCm. On Intel, you use OpenVINO. On Apple silicon you use MLX. All work fine with all the common AI tasks you'd want to do on self-hosted hardware. CUDA was there first and so it has a more mature ecosystem, but, so far, I've found 0 models or tasks I haven't been able to use with ROCm. ll…

The problem with ROCm, unlike CUDA, is that it doesn’t run on much of AMDs own hardware, most notably their iGPU.

Yeah, that kinda sucks, but, all their new generation onboard GPUs are supported by ROCm. e.g. Ryzen AI 395 and 400 series which will be found in mid-to-high end laptops and desktops and motherboards. They seem to have realized that the reason Nvidia is kicking their ass is that people can develop with CUDA on all sorts of hardware, including their personal laptop or desktop.

Re: My first impressions on ROCm and Strix Halo

#25

Post by SwellJoe »

roenxi wrote: post #17
SwellJoe wrote: post #14

Earlier quoted context omitted.

> CUDA solved that problem. CUDA is a proprietary Nvidia product. CUDA solved the problem for Nvidia chips. On AMD GPUs, you use ROCm. On Intel, you use OpenVINO. On Apple silicon you use MLX. All work fine with all the common AI tasks you'd want to do on self-hosted hardware. CUDA was there first and so it has a more mature ecosystem, but, so far, I've found 0 models or tasks I haven't been able to use with ROCm. ll…

Why? What is the point of focusing on something that seems to be a memory management solution when the memory management problem theoretically just went away? That has been one of the big themes in GPU hardware since around 2010 era when AMD committed to ATI. Nvidia tried to solve the memory management problem in the software layer, AMD committed to doing it in hardware. Software was a better bet by around a trillion…

I dunno. GPU programming and performance is above my pay grade. I assume the reason every GPU maker is investing in software is because they understand the problems to be solved and feel it's worth the investment to solve them. I like AMD because their Linux drivers are open source. I like Intel because all their stuff is Open Source. I like Nvidia notably less because none of their stuff is Open Source, not even the Linux drivers.

Re: My first impressions on ROCm and Strix Halo

#26

Post by sidkshatriya »

> It seems that things wouldn't work without a BIOS update: PyTorch was unable to find the GPU. This was easily done on the BIOS settings: it was able to connect to my Wifi network and download it automatically.

Call me traditional but I find it a bit scary for my BIOS to be connecting to WiFi and doing the downloading. Makes me wonder if the new BIOS blob would be secure i.e. did the BIOS connect over securely over https ? Did it check for the appropriate hash/signature etc. ? I would suppose all this is more difficult to do in the BIOS. I would expect better security if this was done in user space in the OS.

I'm much prefer if the OS did the actual downloading followed by the BIOS just doing the installation of the update.

Re: My first impressions on ROCm and Strix Halo

#27

Post by suprjami »

If you are using quants below Q8 then get them from Unsloth or Bartowski.

They are higher quality than the quants you can make yourself due to their imatrix datasets and selective quantisation of different parts of the model.

For Qwen 3.5 Unsloth did 9 terabytes of quants to benchmark the effects of this:

https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

Re: My first impressions on ROCm and Strix Halo

#28

Post by imp0cat »

sidkshatriya wrote: post #26

> It seems that things wouldn't work without a BIOS update: PyTorch was unable to find the GPU. This was easily done on the BIOS settings: it was able to connect to my Wifi network and download it automatically. Call me traditional but I find it a bit scary for my BIOS to be connecting to WiFi and doing the downloading. Makes me wonder if the new BIOS blob would be secure i.e. did the BIOS connect over securely over…

Isn't this pretty much standard in this day and age? HP for example also has this option in BIOS for their laptops (but you still can either download the BIOS blob manually in Linux or use the automatic updater in Windows if you want).

Re: My first impressions on ROCm and Strix Halo

#29

Post by phonon »

fwipsy wrote: post #21
spoaceman7777 wrote: post #10

I'm somewhat confused as to why this is on the front page. It doesn't go into any real detail, and the advice it gives is... not good. You should definitely not be quantizing your own gguf's using an old method like that hf script. There are lots of ways to run LLMs via podman (some even officially recommended by the project!). The chip has been out for almost a year now, and its most notable (and relevant-to-AI) fea…

Quad-channel RAM is common on consumer desktops. Strix Halo has *8* channels, and also very fast RAM (soldered RAM can be faster than dimms because the traces are shorter.)

4 DIMMS =/= 4 channels

Re: My first impressions on ROCm and Strix Halo

#30

Post by sidkshatriya »

imp0cat wrote: post #28
sidkshatriya wrote: post #26

> It seems that things wouldn't work without a BIOS update: PyTorch was unable to find the GPU. This was easily done on the BIOS settings: it was able to connect to my Wifi network and download it automatically. Call me traditional but I find it a bit scary for my BIOS to be connecting to WiFi and doing the downloading. Makes me wonder if the new BIOS blob would be secure i.e. did the BIOS connect over securely over…

Isn't this pretty much standard in this day and age? HP for example also has this option in BIOS for their laptops (but you still can either download the BIOS blob manually in Linux or use the automatic updater in Windows if you want).

> Isn't this pretty much standard in this day and age?

If something is "standard" nowadays does it mean it is the right way to go ?

One of my main issues is that this means your BIOS has to have a WiFi software stack in it, have a TLS stack in it etc. Basically millions of lines of extra code. Most of it in a blob never to be seen by more than a few engineers.

Though in another a way allowing BIOS to perform self updates is good because it doesn't matter if you've installed FreeBSD, OpenBSD, Linux, Windows, you will be able to update your BIOS.

Post reply on HN