Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HipKittens: Fast and furious AMD kernels

hazyresearch.stanford.edu

Post reply on HN
Print view View original

61–70 of 94 posts

Re: HipKittens: Fast and furious AMD kernels

#61

Post by WithinReason »

georgehotz wrote: post #25

Full disclosure, we have a contract with AMD to get Llama 405B training on MI350X on MLPerf. Things are turning around for AMD. If you have an AMD card, go to pytorch.org, click Linux+ROCm and install PyTorch. 3 years ago, this was hopeless. Today, most mainline things work. I ran nanochat on MI300X and it just worked. I think that's true about MI350X now too. The MI350X machine is stable. They are clearly behind NVI…

How far is Tinygrad from being able to represent/search the kind of optimisations listed in the article? i.e.:

  1. data layouts to avoid local memory bank conflicts
  2. read patterns from global memory to optimize L2 cache reuse
  3. warp specialisation
How complex is it to add these into tinygrad?

Re: HipKittens: Fast and furious AMD kernels

#62

Post by ehnto »

electroglyph wrote: post #58
ehnto wrote: post #53

Earlier quoted context omitted.

I run Qwen3 Coder 30b through Ollama on an RTX7900XTX. It works great, I suspect some load gets passed to the 32gb system memory and Ryzen 7 CPU. It's not quite as fast as like Sonnet 4 from an API, but it's really not that bad. It's really great for quick questions so I don't have to google stuff, and it's probably Sonnet4 level of competency at achieving coding tasks. No API served model has been fast enough to rem…

not the best model to use as a showcase, it's blistering fast on anything that isn't a toaster

Great! That's what I am pointing out, it's a 30b param model that fits into an AMD card and runs great. That's what we want.

Re: HipKittens: Fast and furious AMD kernels

#63

Post by christkv »

ehnto wrote: post #53
999900000999 wrote: post #52

With these new developments, are there any implications for getting LLMs running well on consumer AMD chips ? For example, the following laptop which I'm thinking of picking up, has both a strong AMD CPU/IGPU and a RTX 5080. Could we see the AMD side competing with the RTX? I know a dedicated gpu will always be faster though. >HP OMEN MAX 16-ak0003nr 16" Gaming Laptop Computer - Shadow Black Aluminum AMD Ryzen AI 9 H…

I run Qwen3 Coder 30b through Ollama on an RTX7900XTX. It works great, I suspect some load gets passed to the 32gb system memory and Ryzen 7 CPU. It's not quite as fast as like Sonnet 4 from an API, but it's really not that bad. It's really great for quick questions so I don't have to google stuff, and it's probably Sonnet4 level of competency at achieving coding tasks. No API served model has been fast enough to rem…

That's great I have been eyeing a Strix Halo and was wondering how well smaller models are doing. This is great news from the perspective of running local agents.

Re: HipKittens: Fast and furious AMD kernels

#64

Post by vagab0nd »

bratao wrote: post #2

One thing I don't understand about Nvidia’s valuation is that right now a small number of algorithms have 'won,' such as Transformers. The data is very important. Compared to the past where customized code was much more common, such as modeling code and HPC, the ecosystem was very important and it was almost impossible to implement all CUDA and related code. Competitors now only need to optimize for a narrow set of a…

If your competitor has a 5-year lead, and is working as hard as you are, or harder, then you are not gonna catch up any time soon. Also yes networking.

Re: HipKittens: Fast and furious AMD kernels

#65

Post by ACCount37 »

bratao wrote: post #2

One thing I don't understand about Nvidia’s valuation is that right now a small number of algorithms have 'won,' such as Transformers. The data is very important. Compared to the past where customized code was much more common, such as modeling code and HPC, the ecosystem was very important and it was almost impossible to implement all CUDA and related code. Competitors now only need to optimize for a narrow set of a…

By far the easiest way to implement that "small number of algorithms" is with universal number-grinding hardware. Which also protects you against any architectural developments. Hardware takes a damn long time to make.

Re: HipKittens: Fast and furious AMD kernels

#66

Post by slavik81 »

LtdJorge wrote: post #5

Ahh, composable-kernel. The highest offender in the list of software that have produced unrecoverable OOMs in my Gentoo system (it’s actually Clang while compiling CK, which uses upwards of 2.5GB per thread).

I was recently reviewing a CK package for Debian. My test build crashed due to OOM using -j32 on a 64GB workstation, so I tried with -j1 to be safe. That completed successfully after 190 hours!

I think I may need to reduce the number of architectures it's built for to successfully compile it on the official Debian buildd infrastructure, but my (unverified) understanding is that most of its reverse dependencies only need the header-only parts of the library anyway.

I'm told they're working on improving the build times via a few different methods.

Re: HipKittens: Fast and furious AMD kernels

#67

Post by ivape »

0manrho wrote: post #19
wewewedxfgdf wrote: post #4

You'd think AMD would swing in on something like this and fund it with the money needed to succeed. I have no knowledge of it but my guess is no, AMD never misses an opportunity to miss an opportunity - when it comes to GPUs and AI.

> AMD never misses an opportunity to miss an opportunity Well said, their Instinct parts are actually, at a hardware level, very very capable pieces of kit that - ignoring software/dev ecosystem - are very competitive with NVidia. Problem is, AMD has a terrible history of supporting it's hardware (either just outright lack of support, cough Radeon VII; or constantly scrapping things and starting over and thus the eco…

I wouldn’t even look at it like they are learning their lesson. The total addressable market is 1T according to them, and they are usually very conservative with their approach and projections. They will solve the software issue because there is simply too much money in it.

Re: HipKittens: Fast and furious AMD kernels

#68

Post by kieranl »

fulafel wrote: post #40
georgehotz wrote: post #25

Full disclosure, we have a contract with AMD to get Llama 405B training on MI350X on MLPerf. Things are turning around for AMD. If you have an AMD card, go to pytorch.org, click Linux+ROCm and install PyTorch. 3 years ago, this was hopeless. Today, most mainline things work. I ran nanochat on MI300X and it just worked. I think that's true about MI350X now too. The MI350X machine is stable. They are clearly behind NVI…

Does consumer hardware (non-MI) need proprietary kernel drivers for running rocm + pytorch?

No. But you might need a specific version of rocm built for your gpu. These are built on https://github.com/ROCm/TheRock

Right now AI support on AMD is officially only on specific models. But they are working hard to turn this around to have broader support. And making progress.

Re: HipKittens: Fast and furious AMD kernels

#69

Post by fulafel »

999900000999 wrote: post #52

With these new developments, are there any implications for getting LLMs running well on consumer AMD chips ? For example, the following laptop which I'm thinking of picking up, has both a strong AMD CPU/IGPU and a RTX 5080. Could we see the AMD side competing with the RTX? I know a dedicated gpu will always be faster though. >HP OMEN MAX 16-ak0003nr 16" Gaming Laptop Computer - Shadow Black Aluminum AMD Ryzen AI 9 H…

You might think that a dGPU is always faster but the limited memory capacity bites you there (unless you go to datacenter dGPUs that cost tens of thousnds). Look at eg https://www.ywian.com/blog/amd-ryzen-ai-max-plus-395-native-... or the various high end Mac results.

Re: HipKittens: Fast and furious AMD kernels

#70

Post by georgehotz »

WithinReason wrote: post #61
georgehotz wrote: post #25

Full disclosure, we have a contract with AMD to get Llama 405B training on MI350X on MLPerf. Things are turning around for AMD. If you have an AMD card, go to pytorch.org, click Linux+ROCm and install PyTorch. 3 years ago, this was hopeless. Today, most mainline things work. I ran nanochat on MI300X and it just worked. I think that's true about MI350X now too. The MI350X machine is stable. They are clearly behind NVI…

How far is Tinygrad from being able to represent/search the kind of optimisations listed in the article? i.e.: 1. data layouts to avoid local memory bank conflicts 2. read patterns from global memory to optimize L2 cache reuse 3. warp specialisation How complex is it to add these into tinygrad?

1 and 2 are supported, 1 you need to specify, 2 will be found with BEAM. We are working on reimplementing HipKittens in tinygrad, all the stuff is there to do it. See the amd_uop_matmul example.

tinygrad doesn't support 3 yet, it's not needed on any AMD GPUs, and not needed on NVIDIA consumer. It wouldn't be hard to add, but it's important to figure out how it best fits with the existing abstractions. I think everything will eventually move to a more producer-consumer model.

Post reply on HN