Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HipKittens: Fast and furious AMD kernels

hazyresearch.stanford.edu

Post reply on HN
Print view View original

51–60 of 94 posts

Re: HipKittens: Fast and furious AMD kernels

#51

Post by _lyxd »

elteto wrote: post #31
wmobit wrote: post #12

Earlier quoted context omitted.

I'd go so far as to say it's the exact opposite. It's faster and easier to change the hardware than the software.

Counterproof: attempt to modify your graphics card. Then attempt to modify a piece of code. Which one was easier?

You're saying it like hardware and software are disjoint. You design hardware with software in mind (and vice versa); you need to if you want performance rivaling nvidia. This codesign, seeing their products are not only usable but actually tailored to maximize resource utilization in real workloads (not driven by w/e benchmarks), is where AMD seems to lack.

Why oversimplify the premise and frame your take as some 'proof'. Just use the term counter-argument/example

Re: HipKittens: Fast and furious AMD kernels

#52

Post by 999900000999 »

With these new developments, are there any implications for getting LLMs running well on consumer AMD chips ?

For example, the following laptop which I'm thinking of picking up, has both a strong AMD CPU/IGPU and a RTX 5080. Could we see the AMD side competing with the RTX?

I know a dedicated gpu will always be faster though.

>HP OMEN MAX 16-ak0003nr 16" Gaming Laptop Computer - Shadow Black Aluminum AMD Ryzen AI 9 HX 375 (2.0GHz) Processor; NVIDIA GeForce RTX 5080 16GB GDDR7; 32GB DDR5-5600 RAM; 1TB Solid State Drive

Re: HipKittens: Fast and furious AMD kernels

#53

Post by ehnto »

999900000999 wrote: post #52

With these new developments, are there any implications for getting LLMs running well on consumer AMD chips ? For example, the following laptop which I'm thinking of picking up, has both a strong AMD CPU/IGPU and a RTX 5080. Could we see the AMD side competing with the RTX? I know a dedicated gpu will always be faster though. >HP OMEN MAX 16-ak0003nr 16" Gaming Laptop Computer - Shadow Black Aluminum AMD Ryzen AI 9 H…

I run Qwen3 Coder 30b through Ollama on an RTX7900XTX. It works great, I suspect some load gets passed to the 32gb system memory and Ryzen 7 CPU.

It's not quite as fast as like Sonnet 4 from an API, but it's really not that bad.

It's really great for quick questions so I don't have to google stuff, and it's probably Sonnet4 level of competency at achieving coding tasks.

No API served model has been fast enough to remove the urge to do something else while waiting for bigger tasks, so the UX is more or less the same in that regard.

Opencode + ollama + Qwen3 Coder has been a very reasonable alternative to ClaudeCode with Sonnet4.

That is amazing for something running locally.

It is possible that if you actually need AI to be doing all your coding, that you're going to feel differently about the setup. But as a small assistant it's great.

Re: HipKittens: Fast and furious AMD kernels

#54

Post by ehnto »

bratao wrote: post #2

One thing I don't understand about Nvidia’s valuation is that right now a small number of algorithms have 'won,' such as Transformers. The data is very important. Compared to the past where customized code was much more common, such as modeling code and HPC, the ecosystem was very important and it was almost impossible to implement all CUDA and related code. Competitors now only need to optimize for a narrow set of a…

They also don't actually have a moat in the sense that they have patented technology keeping others out of the game. The other chip makers are coming for their lunch eventually.

Re: HipKittens: Fast and furious AMD kernels

#55

Post by latchkey »

ivape wrote: post #45
latchkey wrote: post #32

Earlier quoted context omitted.

As CEO of an AMD NeoCloud for the past 2 years, it is so nice to hear all this and also see the turn around. It is what I bet my business on from the start and I can concur with what George is saying 100%. The out of box experience can be a bit rough around the edges on bleeding edge stuff, but it isn't anything near as bad as it used to be. For example, a month ago nanochat wasn't working well and now it is. The imp…

That’s interesting, I was specifically looking for AMD hardware being offered by neoclouds, they seem to be rare. I like your bet though. The difference between NVDA and AMD has never really existed on a hardware level for decades. AMD has always been on par, and software is software, it will catch up. AMD will be a stock many people will miss because the opportunity has presented itself at the height of AI bubble ta…

You're right, it is a much smaller ecosystem, but I think that is partly intentional as a way to focus efforts and not feed into the bubble, which I feel is a smart move. These are the official partners [0]. I'm Hot Aisle.

George was very smart, $500k in the $90's. I saw it coming even earlier than him, but that's cause I was already aware the hardware was good from my own experiences.

[0] https://www.amd.com/en/products/accelerators/instinct/eval-r...

Re: HipKittens: Fast and furious AMD kernels

#56

Post by georgehotz »

fulafel wrote: post #40
georgehotz wrote: post #25

Full disclosure, we have a contract with AMD to get Llama 405B training on MI350X on MLPerf. Things are turning around for AMD. If you have an AMD card, go to pytorch.org, click Linux+ROCm and install PyTorch. 3 years ago, this was hopeless. Today, most mainline things work. I ran nanochat on MI300X and it just worked. I think that's true about MI350X now too. The MI350X machine is stable. They are clearly behind NVI…

Does consumer hardware (non-MI) need proprietary kernel drivers for running rocm + pytorch?

Nope! Works fine with in-tree somewhat recent kernel. The AMD driver is actually open source, not just a wrapper into a big on device blob like the NVIDIA one. tinygrad also has a driver that doesn't even need the kernel module, just mmapping the PCIe BAR into Python.

Re: HipKittens: Fast and furious AMD kernels

#57

Post by schainks »

AMDAnon wrote: post #14
wewewedxfgdf wrote: post #4

You'd think AMD would swing in on something like this and fund it with the money needed to succeed. I have no knowledge of it but my guess is no, AMD never misses an opportunity to miss an opportunity - when it comes to GPUs and AI.

AMD pays the bare minimum in software to get a product out the door. The company does not even have working performance testing and regressions routinely get shipped to customers. Benchmarks the executives see are ad hoc and not meaningful. HipKittens is an improvement but AMD does not have the ability to understand or track kernel performance so it'll be ignored. This isn't fixable overnight. Company-wide DevOps and…

> We haven't had a fully funded bonus in the past 4+ years.

This is WILD to hear considering how well it appears AMD is executing from the outside.

Re: HipKittens: Fast and furious AMD kernels

#58

Post by electroglyph »

ehnto wrote: post #53
999900000999 wrote: post #52

With these new developments, are there any implications for getting LLMs running well on consumer AMD chips ? For example, the following laptop which I'm thinking of picking up, has both a strong AMD CPU/IGPU and a RTX 5080. Could we see the AMD side competing with the RTX? I know a dedicated gpu will always be faster though. >HP OMEN MAX 16-ak0003nr 16" Gaming Laptop Computer - Shadow Black Aluminum AMD Ryzen AI 9 H…

I run Qwen3 Coder 30b through Ollama on an RTX7900XTX. It works great, I suspect some load gets passed to the 32gb system memory and Ryzen 7 CPU. It's not quite as fast as like Sonnet 4 from an API, but it's really not that bad. It's really great for quick questions so I don't have to google stuff, and it's probably Sonnet4 level of competency at achieving coding tasks. No API served model has been fast enough to rem…

not the best model to use as a showcase, it's blistering fast on anything that isn't a toaster

Re: HipKittens: Fast and furious AMD kernels

#59

Post by jillesvangurp »

bratao wrote: post #2

One thing I don't understand about Nvidia’s valuation is that right now a small number of algorithms have 'won,' such as Transformers. The data is very important. Compared to the past where customized code was much more common, such as modeling code and HPC, the ecosystem was very important and it was almost impossible to implement all CUDA and related code. Competitors now only need to optimize for a narrow set of a…

You are right to question their moat. My view on this is that there's a lot of pressure from essentially all other trillion dollar companies (MS, Google, Amazon, Apple, etc.) to not get locked into a NVidia only ecosystem. Each of those do their own chips. They also use Nvidia but not exclusively. An Android or IOS phone has no nvidia capable chips whatsoever. Neither do most laptops. Apple's M series CPUs don't support it at all typically. And with the exception of some gaming or workstation class laptops, most windows/linux laptops come with either AMD or Intel GPUs. Or lately Qualcomm ARM based architectures with custom GPUs.

Nvidias valuation and moat are centered around data center class GPUs used for training. I don't think they effectively have that space to themselves for much longer. Google is already using their own TPUs at scale for both training and inference. They still use some Nvidia stuff but they seem to be able to keep that off the critical path for anything that needs to run at "Google scale". OpenAI just ordered a bunch of AMD hardware. A lot of AI engineers use Apple laptops that rely on the M series hardware.

In short, the Cuda moat is shrinking. It's still relevant of course and there are a lot of tooling and frameworks that depend on it. That's why everybody still uses it. But not exclusively. And there's a lot of extremely well funded and active development to cut loose from it. AMD of course wants in. So does Intel. And so does everybody else. This HipKittens thing looks like it makes some big steps towards a more neutral software ecosystem.

Re: HipKittens: Fast and furious AMD kernels

#60

Post by mattlondon »

mandelken wrote: post #8
wmf wrote: post #3

Earlier quoted context omitted.

Infiniband is being replaced with UEC (and it isn't needed for inference). For inference there is no moat and smart players are buying/renting AMD or Google TPUs.

I didn't know you can you buy Google TPUs now?

You can pay to use them https://cloud.google.com/tpu
Post reply on HN