Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel Gaudi 3 AI Accelerator

intel.com

Post reply on HN
Print view View original

221–230 of 260 posts

Re: Intel Gaudi 3 AI Accelerator

#221

Post by JonChesterfield »

tucnak wrote: post #129
JonChesterfield wrote: post #95

Earlier quoted context omitted.

We absolutely hammered the MI50 in internal testing for ages. Was solid as far as I can tell. Rocm is sensitive to matching kernel version to driver version to userspace version. Staying very much on the kernel version from a official release and using the corresponding driver is drastically more robust than optimistically mixing different components. In particular, rocm is released and tested as one large blob, and…

I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.

Did you run on the blessed Ubuntu version with the blessed kernel version and the blessed driver version? As otherwise you really are in a development branch.

If you can point me to a repro I'll add it to my todo list. You can probably tag me in the github issue if that's where you reported it.

Re: Intel Gaudi 3 AI Accelerator

#222

Post by treprinum »

rhelz wrote: post #166
JonChesterfield wrote: post #115

Earlier quoted context omitted.

I really like this side to AMD. There's a strategic call somewhere high up to bias towards collaboration with other companies. Sharing the fabric specifications with broadcom was an amazing thing to see. It's not out of the question that we'll see single chips with chiplets made by different companies attached together.

Well, lets not forget, AMD is AMD because they reverse-engineered Intel chips....

IBM didn't want to rely solely on Intel when introducing PCs so it forced Intel to share its arch with another manufacturer that turned out to be AMD. It's not like AMD stole it. Math coprocessor was in turn invented by AMD (Am9511, Am9512) and licensed by Intel (8231, 8232).

Re: Intel Gaudi 3 AI Accelerator

#223

Post by JonChesterfield »

michaelt wrote: post #215
FeepingCreature wrote: post #194

Earlier quoted context omitted.

Same here with SD on 7900XTX. Most of the time for me it's sufficient to reset the card with rocm-smi --gpureset -d 0.

Only "most of the time" ? :( You'd hope at $15,000+ per unit, you wouldn't have to reset it at all...

It's $1000 per, no? This is one of the gaming cards.

Re: Intel Gaudi 3 AI Accelerator

#225

Post by mike_hearn »

gymbeaux wrote: post #200
mk_stjames wrote: post #111

One nice thing about this (and the new offerings from AMD) is that they will be using the "open accelerator module (OAM)" interface- which standardizes the connector that they use to put them on baseboards, similar to the SXM connections of Nvidia that use MegArray connectors to thier baseboards. With Nvidia, the SXM connection pinouts have always been held proprietary and confidential. For example, P100's and V100's…

As “humble” as NVIDIA’s CEO appears to be, NVIDIA the company (he’s been running this whole time), made decision after decision with the simple intention of killing off its competition (ATI/AMD). Gameworks is my favorite example- essentially if you wanted a video game to look as good as possible, you needed an NVIDIA GPU. Those same games played on AMD GPUs just didn’t look as good. Now that video gaming is secondary…

You don't need to use C++ to interface with CUDA or even write it.

A while ago NVIDIA and the GraalVM team demoed grCUDA which makes it easy to share memory with CUDA kernels and invoke them from any managed language that runs on GraalVM (which includes JIT compiled Python). Because it's integrated with the compiler the invocation overhead is low:

https://developer.nvidia.com/blog/grcuda-a-polyglot-language...

And TornadoVM lets you write kernels in JVM langs that are compiled through to CUDA:

https://www.tornadovm.org

There are similar technologies for other languages/runtimes too. So I don't think that will cause NVIDIA to lose ground.

Re: Intel Gaudi 3 AI Accelerator

#226

Post by egorfine »

gymbeaux wrote: post #199
kkielhofner wrote: post #189

Earlier quoted context omitted.

Pascal series are cheap because they are CUDA compute capability 6.0 and lack Tensor Cores. Volta (7.0) was the first to have Tensor Cores and in many cases is the bare minimum for modern/current stacks. See flash attention, triton, etc as core enabling libraries. Not to mention all of the custom CUDA kernels all over the place. Take all of this and then stack layers on top of them... Unfortunately there is famously…

Hey that’s not fair, the X5690 is VERY efficient… at heating a home in the winter time.

Easier said than done. I've got a dual X5690 at home in Kiev, Ukraine and I just couldn't find anything to run on it 24x7. And it doesn't produce much heat idling. I mean at all.

Re: Intel Gaudi 3 AI Accelerator

#227

Post by egorfine »

narrator wrote: post #143
wongarsu wrote: post #35

Earlier quoted context omitted.

Probably not. An 40GB Nvidia A100 is arguably reasonable for a workstation at $6000. Depending on your definition an 80GB A100 for $16000 is still reasonable. I don't see this being cheaper than an 80GB A100. Probably a good bit more expensive, seeing as it has more RAM, compares itself favorably to the H100, and has enough compelling features that it probably doesn't have to (strongly) compete on price.

Isn't it much better to get a Mac Studio with an M2 Max and 192gb of Ram and 31 terraflops for $6599 and run llama.cpp?

For LLM inference - yes absolutely.

Re: Intel Gaudi 3 AI Accelerator

#228

Post by Aissen »

JonChesterfield wrote: post #221
tucnak wrote: post #129

Earlier quoted context omitted.

I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.

Did you run on the blessed Ubuntu version with the blessed kernel version and the blessed driver version? As otherwise you really are in a development branch. If you can point me to a repro I'll add it to my todo list. You can probably tag me in the github issue if that's where you reported it.

> blessed Ubuntu version with the blessed kernel version

To an SRE, this is a nightmare to read. Cuda is bad in this regard (can often prevent major kernel version updates), but this is worse.

Re: Intel Gaudi 3 AI Accelerator

#229

Post by gymbeaux »

egorfine wrote: post #226
gymbeaux wrote: post #199

Earlier quoted context omitted.

Hey that’s not fair, the X5690 is VERY efficient… at heating a home in the winter time.

Easier said than done. I've got a dual X5690 at home in Kiev, Ukraine and I just couldn't find anything to run on it 24x7. And it doesn't produce much heat idling. I mean at all.

All the sane and rational people are rooting for you here in the U.S. I’m sorry our government is garbage and aid hasn’t been coming through as expected. Hopefully Ukraine can stick it to that chicken-fucker in the Kremlin and retake Crimea too.

I didn’t have an X5690 because the TDP was too high for my server’s heatsinks, but I had 90W variants of the same generation. To me, two at idle produced noticeable heat, though not as much as four idling in a PowerEdge R910 did. The R910 idled at around 300W.

There’s always Folding@Home if you don’t mind the electric bill. Plex is another option. I know a guy running a massive Plex server that was on Westmere/Nehalem Xeons until I gave him my R720 with Haswell Xeons.

Re: Intel Gaudi 3 AI Accelerator

#230

Post by gymbeaux »

mike_hearn wrote: post #225
gymbeaux wrote: post #200

Earlier quoted context omitted.

As “humble” as NVIDIA’s CEO appears to be, NVIDIA the company (he’s been running this whole time), made decision after decision with the simple intention of killing off its competition (ATI/AMD). Gameworks is my favorite example- essentially if you wanted a video game to look as good as possible, you needed an NVIDIA GPU. Those same games played on AMD GPUs just didn’t look as good. Now that video gaming is secondary…

You don't need to use C++ to interface with CUDA or even write it. A while ago NVIDIA and the GraalVM team demoed grCUDA which makes it easy to share memory with CUDA kernels and invoke them from any managed language that runs on GraalVM (which includes JIT compiled Python). Because it's integrated with the compiler the invocation overhead is low: https://developer.nvidia.com/blog/grcuda-a-polyglot-language... And To…

So these alternatives exist yes, but are they “production ready”- in other words, are they being used. My opinion is that while you can use another language, most companies for one reason or another are still using C++. I just don’t really know what the reason(s) are.

I think about other areas in tech where you can use whatever language, but it isn’t practical to do so. I can write a backend API server in Swift… or perhaps more relevant- I can use AMD’s ROCm to do… anything.

Post reply on HN