Intel Gaudi 3 AI Accelerator • Hacker News Board

Re: Intel Gaudi 3 AI Accelerator

#191

Post by buildbot » Wed, Apr 10, 2024, 4:50 AM UTC

mk_stjames wrote: post #172

buildbot wrote: post #138

Earlier quoted context omitted.

The SXM2 interface is actually publicly documented! There is an open compute spec for a 8-way baseboard. You can find the pinouts there.

Upon further review... I think any actual base board schematics / pinouts touching the Nvidia hardware directly is indeed kept behind some sort of NDA or OEM license agreement and is specifically kept out of any of those documents for the Open Compute project JBOG rigs. I think this is literally the impetus for their OAM spec which makes the pinout open and shareable. Up until that, they had to keep the actual design…

Hmm interesting, I was linked to an OCP dropbox with a version that did have the connector pinouts. Maybe something someone shouldn’t have posted then…

Re: Intel Gaudi 3 AI Accelerator

#192

Post by ukuina » Wed, Apr 10, 2024, 5:04 AM UTC

brookst wrote: post #110

whalesalad wrote: post #71

Earlier quoted context omitted.

to be fair intel is not known for naming things well.

Yeah I can't believe people are nitpicking the name when it could just as easily have been AIX19200xvr4200AI.

Assuming that's going to be the datasheet naming.

Re: Intel Gaudi 3 AI Accelerator

#193

Post by FeepingCreature » Wed, Apr 10, 2024, 5:16 AM UTC

ZoomerCretin wrote: post #148

JonChesterfield wrote: post #118

Earlier quoted context omitted.

All the dev work at AMD is on our own hardware. Even things like the corporate laptops are ryzen based. The first gen ryzen laptop I got was terrible but it wasn't intel. We also do things like develop ROCm on the non-qualified cards and build our tools with our tools. It would be crazy not to.

Yes that's why I qualified "serious" dogfooding. Of course you use your hardware for your own development work, but it's clearly not enough given that showstopper driver issues are going unfixed for half a year.

Way more than half a year. The 7900XTX came out two years ago and still hits hardware resets with Stable Diffusion.

Re: Intel Gaudi 3 AI Accelerator

#194

Post by FeepingCreature » Wed, Apr 10, 2024, 5:17 AM UTC

tucnak wrote: post #129

JonChesterfield wrote: post #95

Earlier quoted context omitted.

We absolutely hammered the MI50 in internal testing for ages. Was solid as far as I can tell. Rocm is sensitive to matching kernel version to driver version to userspace version. Staying very much on the kernel version from a official release and using the corresponding driver is drastically more robust than optimistically mixing different components. In particular, rocm is released and tested as one large blob, and…

I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.

Same here with SD on 7900XTX. Most of the time for me it's sufficient to reset the card with rocm-smi --gpureset -d 0.

Re: Intel Gaudi 3 AI Accelerator

#195

Post by wmf » Wed, Apr 10, 2024, 5:23 AM UTC

sandGorgon wrote: post #188

> Intel Gaudi software integrates the PyTorch framework and provides optimized Hugging Face community-based models – the most-common AI framework for GenAI developers today. This allows GenAI developers to operate at a high abstraction level for ease of use and productivity and ease of model porting across hardware types. what is the programming interface here ? this is not CUDA right ...so how is this being done ?

PyTorch has a bunch of backends including CUDA, ROCm, OneAPI, etc.

Re: Intel Gaudi 3 AI Accelerator

#196

Post by abdullin » Wed, Apr 10, 2024, 5:24 AM UTC

KeplerBoy wrote: post #190

algo_trader wrote: post #162

Earlier quoted context omitted.

Somewhat off topic: Is there a similar "magic value card" for low memory (2GB?) 8-bit LLMs? Since memory is the expensive bit, surely there are low cost low memory models?

I believe that's what tenstorrent is aiming for.

The main offer of Tenstorrent goes into server racks and is designed to form clusters.

Standalone cards are more like dev kits.

(I’ve been tracking Tenstorrent for 3+ years and currently have Grayskull in ML test rig together with 3090)

Re: Intel Gaudi 3 AI Accelerator

#197

Post by wmf » Wed, Apr 10, 2024, 5:26 AM UTC

amelius wrote: post #136

Missing in these pictures are the thermal management solutions.

It's going to look very similar to an Nvidia SXM or AMD MI300 heatsink since these all have similar form factors.

Re: Intel Gaudi 3 AI Accelerator

#198

Post by tmikaeld » Wed, Apr 10, 2024, 5:37 AM UTC

neilmovva wrote: post #12

A bit surprised that they're using HBM2e, which is what Nvidia A100 (80GB) used back in 2020. But Intel is using 8 stacks here, so Gaudi 3 achieves comparable total bandwidth (3.7TB/s) to H100 (3.4TB/s) which uses 5 stacks of HBM3. Hopefully the older HBM has better supply - HBM3 is hard to get right now! The Gaudi 3 multi-chip package also looks interesting. I see 2 central compute dies, 8 HBM die stacks, and then 6…

I was just about to comment on this, apparently all production capacity for hbm is tapped out until early 2026

Re: Intel Gaudi 3 AI Accelerator

#199

Post by gymbeaux » Wed, Apr 10, 2024, 5:39 AM UTC

kkielhofner wrote: post #189

mk_stjames wrote: post #111

One nice thing about this (and the new offerings from AMD) is that they will be using the "open accelerator module (OAM)" interface- which standardizes the connector that they use to put them on baseboards, similar to the SXM connections of Nvidia that use MegArray connectors to thier baseboards. With Nvidia, the SXM connection pinouts have always been held proprietary and confidential. For example, P100's and V100's…

Pascal series are cheap because they are CUDA compute capability 6.0 and lack Tensor Cores. Volta (7.0) was the first to have Tensor Cores and in many cases is the bare minimum for modern/current stacks. See flash attention, triton, etc as core enabling libraries. Not to mention all of the custom CUDA kernels all over the place. Take all of this and then stack layers on top of them... Unfortunately there is famously…

Hey that’s not fair, the X5690 is VERY efficient… at heating a home in the winter time.

Re: Intel Gaudi 3 AI Accelerator

#200

Post by gymbeaux » Wed, Apr 10, 2024, 5:46 AM UTC

mk_stjames wrote: post #111

One nice thing about this (and the new offerings from AMD) is that they will be using the "open accelerator module (OAM)" interface- which standardizes the connector that they use to put them on baseboards, similar to the SXM connections of Nvidia that use MegArray connectors to thier baseboards. With Nvidia, the SXM connection pinouts have always been held proprietary and confidential. For example, P100's and V100's…

As “humble” as NVIDIA’s CEO appears to be, NVIDIA the company (he’s been running this whole time), made decision after decision with the simple intention of killing off its competition (ATI/AMD). Gameworks is my favorite example- essentially if you wanted a video game to look as good as possible, you needed an NVIDIA GPU. Those same games played on AMD GPUs just didn’t look as good.

Now that video gaming is secondary (tertiary?) to Nvidia’s revenue stream, they could give a shit which brand gamers prefer. It’s small time now. All that matters is who companies are buying their GPUs from for AI stuff. Break down that CUDA wall and it’s open-season. I wonder how they plan to stave that off. It’s only a matter of time before people get tired of writing C++ code to interface with CUDA.