Intel Gaudi 3 AI Accelerator • Hacker News Board

Re: Intel Gaudi 3 AI Accelerator

#51

Post by chessgecko » Tue, Apr 09, 2024, 6:10 PM UTC

wongarsu wrote: post #35

sairahul82 wrote: post #18

Can we expect the price of 'Gaudi 3 PCIe' to be reasonable enough to put in a workstation? That would be a game changer for local LLMs

Probably not. An 40GB Nvidia A100 is arguably reasonable for a workstation at $6000. Depending on your definition an 80GB A100 for $16000 is still reasonable. I don't see this being cheaper than an 80GB A100. Probably a good bit more expensive, seeing as it has more RAM, compares itself favorably to the H100, and has enough compelling features that it probably doesn't have to (strongly) compete on price.

I think you're right on the price, but just to give some false hope. I think newish hbm (and this is hbm2e which is a little older) is around $15/gb so for 128 gb thats $1920. There are some other cogs, but in theory they could sell this for like $3-4k and make some gross profit while getting some hobbyist mindshare/research code written for it. I doubt they will though, it might eat too much into profits from the non pcie variants.

Re: Intel Gaudi 3 AI Accelerator

#52

Post by dist-epoch » Tue, Apr 09, 2024, 6:11 PM UTC

colechristensen wrote: post #9

Anyone have experience and suggestions for an AI accelerator? Think prototype consumer product with total cost preferably < $500, definitely less than $1000.

All new CPUs will have so called NPUs inside them. For helping running models locally.

Re: Intel Gaudi 3 AI Accelerator

#53

Post by kylixz » Tue, Apr 09, 2024, 6:11 PM UTC

This is a bit snarky — but will Intel actually keep this product line alive for more than a few years? Having been bitten by building products around some of their non-x86 offerings where they killed good IP off and then failed to support it… I’m skeptical.

I truly do hope it is successful so we can have some alternative accelerators.

Re: Intel Gaudi 3 AI Accelerator

#54

Post by Workaccount2 » Tue, Apr 09, 2024, 6:12 PM UTC

wongarsu wrote: post #35

sairahul82 wrote: post #18

Can we expect the price of 'Gaudi 3 PCIe' to be reasonable enough to put in a workstation? That would be a game changer for local LLMs

Probably not. An 40GB Nvidia A100 is arguably reasonable for a workstation at $6000. Depending on your definition an 80GB A100 for $16000 is still reasonable. I don't see this being cheaper than an 80GB A100. Probably a good bit more expensive, seeing as it has more RAM, compares itself favorably to the H100, and has enough compelling features that it probably doesn't have to (strongly) compete on price.

Interestingly they are using HBME2 memory which is a few years old at this point. The price might end up being surprisingly good because of this.

Re: Intel Gaudi 3 AI Accelerator

#55

Post by AnonMO » Tue, Apr 09, 2024, 6:13 PM UTC

YetAnotherNick wrote: post #28

So now hardware companies stopped reporting FLOP/s number and reports in arbitrary unit of parallel operation/s.

1835 tflops fp8. you have to look for it, but they posted it. The link in the op is just an announcement. the white paper has more info. https://www.intel.com/content/www/us/en/content-details/8174...

Re: Intel Gaudi 3 AI Accelerator

#56

Post by AnonMO » Tue, Apr 09, 2024, 6:16 PM UTC

it's crazy that Intel can't manufacture its own chips atm, but it looks like that might change in the coming years as new fabs come online.

Re: Intel Gaudi 3 AI Accelerator

#57

Post by alecco » Tue, Apr 09, 2024, 6:25 PM UTC

Gaudi 3 has PCIe 4.0 (vs. H100 PCIe 5.0, so 2x the bandwidth). Probably not a deal-breaker but it's strange for Intel (of all vendors) to lag behind in PCIe.

Re: Intel Gaudi 3 AI Accelerator

#58

Post by 0cf8612b2e1e » Tue, Apr 09, 2024, 6:32 PM UTC

wongarsu wrote: post #35

sairahul82 wrote: post #18

Can we expect the price of 'Gaudi 3 PCIe' to be reasonable enough to put in a workstation? That would be a game changer for local LLMs

Probably not. An 40GB Nvidia A100 is arguably reasonable for a workstation at $6000. Depending on your definition an 80GB A100 for $16000 is still reasonable. I don't see this being cheaper than an 80GB A100. Probably a good bit more expensive, seeing as it has more RAM, compares itself favorably to the H100, and has enough compelling features that it probably doesn't have to (strongly) compete on price.

Surely NVidia’s pricing is more what the market will bear vs an intrinsic cost to build. Intel being the underdog should be willing to offer a discount just to get their foot in the door.

Re: Intel Gaudi 3 AI Accelerator

#59

Post by wmf » Tue, Apr 09, 2024, 6:35 PM UTC

0cf8612b2e1e wrote: post #58

wongarsu wrote: post #35

Earlier quoted context omitted.

Probably not. An 40GB Nvidia A100 is arguably reasonable for a workstation at $6000. Depending on your definition an 80GB A100 for $16000 is still reasonable. I don't see this being cheaper than an 80GB A100. Probably a good bit more expensive, seeing as it has more RAM, compares itself favorably to the H100, and has enough compelling features that it probably doesn't have to (strongly) compete on price.

Surely NVidia’s pricing is more what the market will bear vs an intrinsic cost to build. Intel being the underdog should be willing to offer a discount just to get their foot in the door.

Nvidia is charging $35K so a discount relative to that is still very expensive.

Re: Intel Gaudi 3 AI Accelerator

#60

Post by wmf » Tue, Apr 09, 2024, 6:37 PM UTC

alecco wrote: post #57

Gaudi 3 has PCIe 4.0 (vs. H100 PCIe 5.0, so 2x the bandwidth). Probably not a deal-breaker but it's strange for Intel (of all vendors) to lag behind in PCIe.

N5, PCIe 4.0, and HBM2e. This chip was probably delayed two years.