Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HipKittens: Fast and furious AMD kernels

hazyresearch.stanford.edu

Post reply on HN
Print view View original

21–30 of 94 posts

Re: HipKittens: Fast and furious AMD kernels

#21

Post by BNE »

AMDAnon wrote: post #14
wewewedxfgdf wrote: post #4

You'd think AMD would swing in on something like this and fund it with the money needed to succeed. I have no knowledge of it but my guess is no, AMD never misses an opportunity to miss an opportunity - when it comes to GPUs and AI.

AMD pays the bare minimum in software to get a product out the door. The company does not even have working performance testing and regressions routinely get shipped to customers. Benchmarks the executives see are ad hoc and not meaningful. HipKittens is an improvement but AMD does not have the ability to understand or track kernel performance so it'll be ignored. This isn't fixable overnight. Company-wide DevOps and…

> Teams with good leadership maintain their own shadow IT teams.

Yes, this is true. Painfully true.

Re: HipKittens: Fast and furious AMD kernels

#22

Post by AMDAnon »

0manrho wrote: post #19
wewewedxfgdf wrote: post #4

You'd think AMD would swing in on something like this and fund it with the money needed to succeed. I have no knowledge of it but my guess is no, AMD never misses an opportunity to miss an opportunity - when it comes to GPUs and AI.

> AMD never misses an opportunity to miss an opportunity Well said, their Instinct parts are actually, at a hardware level, very very capable pieces of kit that - ignoring software/dev ecosystem - are very competitive with NVidia. Problem is, AMD has a terrible history of supporting it's hardware (either just outright lack of support, cough Radeon VII; or constantly scrapping things and starting over and thus the eco…

> constantly scrapping things and starting over and thus the ecosystem never matured

AMD hires talented people at below-market and doesn't promote them or give raises. This causes employees to aim at resume-driven development by reinventing the wheel so they can get a job somewhere else.

It's a similar problem to Google, except at Google it's because promotions are explicitly for people that ship new products.

Re: HipKittens: Fast and furious AMD kernels

#23

Post by BNE »

0manrho wrote: post #19
wewewedxfgdf wrote: post #4

You'd think AMD would swing in on something like this and fund it with the money needed to succeed. I have no knowledge of it but my guess is no, AMD never misses an opportunity to miss an opportunity - when it comes to GPUs and AI.

> AMD never misses an opportunity to miss an opportunity Well said, their Instinct parts are actually, at a hardware level, very very capable pieces of kit that - ignoring software/dev ecosystem - are very competitive with NVidia. Problem is, AMD has a terrible history of supporting it's hardware (either just outright lack of support, cough Radeon VII; or constantly scrapping things and starting over and thus the eco…

Our hardware is arguably better (spec for spec) apart from critical areas like memory bandwidth, and GPU to GPU bandwidth. You can tweak your implementations to get the same if not better performance. We do that, we see this, our customers see this.

ROCM pre Rock, suffers from the ossification in the engineering organization. The Rock seeks to completely change that, and the team driving it is amazing. Try out the pre-alpha installer. It is already better than the default installer.

There is hope.

Re: HipKittens: Fast and furious AMD kernels

#24

Post by [deleted] »

BNE wrote: post #23
0manrho wrote: post #19

Earlier quoted context omitted.

> AMD never misses an opportunity to miss an opportunity Well said, their Instinct parts are actually, at a hardware level, very very capable pieces of kit that - ignoring software/dev ecosystem - are very competitive with NVidia. Problem is, AMD has a terrible history of supporting it's hardware (either just outright lack of support, cough Radeon VII; or constantly scrapping things and starting over and thus the eco…

Our hardware is arguably better (spec for spec) apart from critical areas like memory bandwidth, and GPU to GPU bandwidth. You can tweak your implementations to get the same if not better performance. We do that, we see this, our customers see this. ROCM pre Rock, suffers from the ossification in the engineering organization. The Rock seeks to completely change that, and the team driving it is amazing. Try out the pr…

[deleted]

Re: HipKittens: Fast and furious AMD kernels

#25

Post by georgehotz »

Full disclosure, we have a contract with AMD to get Llama 405B training on MI350X on MLPerf.

Things are turning around for AMD. If you have an AMD card, go to pytorch.org, click Linux+ROCm and install PyTorch. 3 years ago, this was hopeless. Today, most mainline things work. I ran nanochat on MI300X and it just worked. I think that's true about MI350X now too. The MI350X machine is stable.

They are clearly behind NVIDIA, nobody doubts that. And a lot of investment into software will be required to catch up, ecosystem, compiler, and driver. But 2 years ago they seemed hopeless, now they don't. Things take time. HipKittens is a great codebase to study to see where AMD's LLVM backend is still lacking; compare it to the CUDA Kittens.

For training, it's NVIDIA and Google in first. AMD in second. And nobody in third. Intel and Tenstorrent are not remotely close. Huawei examples segfaulted. Groq gave up selling chips. Cerebras isn't available anywhere. Trainium had a 5 day wait time to get one instance and I lost interest.

Re: HipKittens: Fast and furious AMD kernels

#26

Post by wmf »

patagurbon wrote: post #20
wmf wrote: post #3

Earlier quoted context omitted.

Infiniband is being replaced with UEC (and it isn't needed for inference). For inference there is no moat and smart players are buying/renting AMD or Google TPUs.

Do you have evidence for this? I don’t think Nvidia is switching to Ultra Ethernet, just adding it to the product line-up

Sorry, I don't mean Nvidia is adopting UEC (they probably hate it). I should have said UEC can substitute for Infiniband.

Re: HipKittens: Fast and furious AMD kernels

#27

Post by suprjami »

wewewedxfgdf wrote: post #4

You'd think AMD would swing in on something like this and fund it with the money needed to succeed. I have no knowledge of it but my guess is no, AMD never misses an opportunity to miss an opportunity - when it comes to GPUs and AI.

AMD have had people contribute optimised ROCm kernels in the past. They closed the PR without merge. ROCm are not interested in this. Baffling behaviour.

Re: HipKittens: Fast and furious AMD kernels

#29

Post by Aurornis »

AMDAnon wrote: post #17
observationist wrote: post #16

Earlier quoted context omitted.

The MBAs are in charge, and now AMD is the new Intel? It's not only not fixable overnight, but it's not fixable at all if the leadership thinks they can coast on simply being not as bad as Intel, and Intel has a helluva lot of inertia and ability to simply sell OEM units on autopilot. Sounds like the AMD board needs to get their heads out of their asses and shake up leadership.

The MBAs have always been in charge to an extent. But the real issue is we don't want to invest in beating Nvidia on quality. Otherwise we wouldn't be doing stock buybacks and instead use the money on poaching engineers. The mindset is that we maintain a comfortable second place by creating a shittier but cheaper product. That is how AMD has operated since 1959 as a second source to Fairchild Semiconductor and Intel.…

> What they should do is bump TC by 60-70% and simultaneously lay off 50% of the engineers.

I was part of company with a similar problem. If AMD’s situation is similar to what I dealt with, it’s more complicated. When you start doing deep cut layoffs at the IC level combined with expectations of big salary increases for those who remain, the office politics escalate to a level I didn’t know was possible.

All of those people who do nothing find a way to join forces with those people who are showing those inflated benchmarks to execs and before you know it the layoffs are about as accurate as random chance when it comes to cutting the dead weight from the company.

In my experience, the change needs to start closer to the top: Upper layers of management need to be shaken up. Middle management audited by new upper management hires who have fresh eyes and aren’t afraid to make honest evaluations. High performing teams who are stuck under management hell need to be identified and rotated into other projects that are critical for the company but have become occupied by fiefdom-building managers. Hiring needs to ramp up to bring in new talent that was previously priced out by the low comp.

It’s hard. I wish there was an easy way to cut the low performers, but they have an amazing way of teaming up with the bad managers. Maybe because they have so much free time to do office politics because they’re not doing much work.

Re: HipKittens: Fast and furious AMD kernels

#30

Post by mountainriver »

bratao wrote: post #2

One thing I don't understand about Nvidia’s valuation is that right now a small number of algorithms have 'won,' such as Transformers. The data is very important. Compared to the past where customized code was much more common, such as modeling code and HPC, the ecosystem was very important and it was almost impossible to implement all CUDA and related code. Competitors now only need to optimize for a narrow set of a…

Transformers aren’t really one thing, the way they are implemented is wildly different. If it wasn’t then vllm and TRL would be easy
Post reply on HN