Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel Gaudi 3 AI Accelerator

intel.com

Post reply on HN
Print view View original

181–190 of 260 posts

Re: Intel Gaudi 3 AI Accelerator

#181

Post by dfex »

astrodust wrote: post #117
chucke1992 wrote: post #98

Earlier quoted context omitted.

Q6600 was quite good but E8400 was the best.

Q6600 is the spiritual successor to the ABIT BP6 Dual Celeron option: https://en.wikipedia.org/wiki/ABIT_BP6

The ABit BP6 bought me so much "cred" at LAN Parties back in the day - the only dual socket motherboard in the building, and paired with two Creative Voodoo 2 GPUs in SLI mode, that thing was a beast (for the late nineties).

I seem to recall that only Quake 2 or 3 was capable of actually using that second processor during a game, but that wasn't the point ;)

Re: Intel Gaudi 3 AI Accelerator

#182

Post by doublepg23 »

magic_hamster wrote: post #167
narrator wrote: post #143

Earlier quoted context omitted.

Isn't it much better to get a Mac Studio with an M2 Max and 192gb of Ram and 31 terraflops for $6599 and run llama.cpp?

Macs don't support CUDA which means all that wonderful hardware will be useless when trying to do anything with AI for at least a few years. There's Metal but it has its own set of problems, biggest one being it isn't a drop in CUDA replacement.

I'm assuming this won't support CUDA either?

Re: Intel Gaudi 3 AI Accelerator

#183

Post by pavelstoev »

mk_stjames wrote: post #111

One nice thing about this (and the new offerings from AMD) is that they will be using the "open accelerator module (OAM)" interface- which standardizes the connector that they use to put them on baseboards, similar to the SXM connections of Nvidia that use MegArray connectors to thier baseboards. With Nvidia, the SXM connection pinouts have always been held proprietary and confidential. For example, P100's and V100's…

Best Tflops/$ is actually 4090, then 3090. Also L4

Re: Intel Gaudi 3 AI Accelerator

#184

Post by modeless »

ac29 wrote: post #173
InvestorType wrote: post #164

This appears to be manufactured by TSMC (or Samsung). The press release says it will use a 5nm process, which is not on Intel's roadmap. "The Intel Gaudi 3 accelerator, architected for efficient large-scale AI compute, is manufactured on a 5 nanometer (nm) process"

Habana was an acquisition and their use of TSMC predates the acquisition.

Yeah, but if Intel can't even get internal customers to adopt their foundry services it seems to bode poorly for the future of the company.

Re: Intel Gaudi 3 AI Accelerator

#185

Post by 7speter »

KeplerBoy wrote: post #131
formerly_proven wrote: post #130

Earlier quoted context omitted.

The price is low because they’re useless (except for replacing dead cards in a DGX), if you had a 40$ PCIe AIC-to-SXM adapter, the price would go up a lot. > I'm one of those people who finds 'retro-super-computing' a cool hobby and thus the interfaces like OAM being open means that these devices may actually have a life for hobbyists in 8~10 years instead of being sent directly to the bins due to secret interfaces a…

I'm not sure the prices would go up that much. What would anyone buy that card for? Yes, it has a decent memory bandwidth (~750 GB/s) and it runs CUDA. But it only has 16 GB and doesn't support tensor cores or low precision floats. It's in a weird place.

The pci-e p100 is has 16gb vram and won’t go below 160 dollars. Prices for these things would pick up if you could put them in some sort of pcie adapter

Re: Intel Gaudi 3 AI Accelerator

#186

Post by 7speter »

_ea1k wrote: post #161
KeplerBoy wrote: post #131

Earlier quoted context omitted.

I'm not sure the prices would go up that much. What would anyone buy that card for? Yes, it has a decent memory bandwidth (~750 GB/s) and it runs CUDA. But it only has 16 GB and doesn't support tensor cores or low precision floats. It's in a weird place.

IDK, is it really that much more powerful than the P40, which is already fairly cheap?

Well, the p40 has 24gb VRAM, which makes it the perfect hobbyist card for a llm, assuming you can keep it cool.

Re: Intel Gaudi 3 AI Accelerator

#188

Post by sandGorgon »

>Intel Gaudi software integrates the PyTorch framework and provides optimized Hugging Face community-based models – the most-common AI framework for GenAI developers today. This allows GenAI developers to operate at a high abstraction level for ease of use and productivity and ease of model porting across hardware types.

what is the programming interface here ? this is not CUDA right ...so how is this being done ?

Re: Intel Gaudi 3 AI Accelerator

#189

Post by kkielhofner »

mk_stjames wrote: post #111

One nice thing about this (and the new offerings from AMD) is that they will be using the "open accelerator module (OAM)" interface- which standardizes the connector that they use to put them on baseboards, similar to the SXM connections of Nvidia that use MegArray connectors to thier baseboards. With Nvidia, the SXM connection pinouts have always been held proprietary and confidential. For example, P100's and V100's…

Pascal series are cheap because they are CUDA compute capability 6.0 and lack Tensor Cores. Volta (7.0) was the first to have Tensor Cores and in many cases is the bare minimum for modern/current stacks.

See flash attention, triton, etc as core enabling libraries. Not to mention all of the custom CUDA kernels all over the place. Take all of this and then stack layers on top of them...

Unfortunately there is famously "GPU poor vs GPU rich". Pascal puts you at "GPU destitute" (regardless of assembled VRAM) and outside of implementations like llama.cpp that go incredible and impressive lengths to support these old archs you will very quickly run into show-stopping issues that make you wish you just handed over the money for >= 7.0.

I support any use of old hardware but this kind of reminds me of my "ancient" X5690 that has impressive performance (relatively speaking) but always bites me because it doesn't have AVX.

Re: Intel Gaudi 3 AI Accelerator

#190

Post by KeplerBoy »

algo_trader wrote: post #162
KeplerBoy wrote: post #139

Earlier quoted context omitted.

Only if the specific workload needs FP64 (4.5 Tflop/s), the 9 Tflop/s for FP32 can be had for cheap with Turing or Ampere consumer cards. Still, your point stands. It's crazy how that 2016 GPU has two thirds the FP32 power of this new 2024 unobtanium card and infinitely more FP64.

Somewhat off topic: Is there a similar "magic value card" for low memory (2GB?) 8-bit LLMs? Since memory is the expensive bit, surely there are low cost low memory models?

I believe that's what tenstorrent is aiming for.
Post reply on HN