Intel Gaudi 3 AI Accelerator
intel.com
Intel Gaudi 3 AI Accelerator
1–10 of 260 posts
Re: Intel Gaudi 3 AI Accelerator
#2> Memory Boost for LLM Capacity Requirements: 128 gigabytes (GB) of HBMe2 memory capacity, 3.7 terabytes (TB) of memory bandwidth ...
I didn't know "terabytes (TB)" was a unit of memory bandwidth...
Re: Intel Gaudi 3 AI Accelerator
#3> Memory Boost for LLM Capacity Requirements: 128 gigabytes (GB) of HBMe2 memory capacity, 3.7 terabytes (TB) of memory bandwidth ... I didn't know "terabytes (TB)" was a unit of memory bandwidth...
It’s equivalent to about thirteen football fields per arn if that helps.
Re: Intel Gaudi 3 AI Accelerator
#4128GB in one chip seems important with the rise of sparse architectures like MoE. Hopefully these are competitive with Nvidia's offerings, though in the end they will be competing for the same fab space as Nvidia if I'm not mistaken.
Re: Intel Gaudi 3 AI Accelerator
#5> Memory Boost for LLM Capacity Requirements: 128 gigabytes (GB) of HBMe2 memory capacity, 3.7 terabytes (TB) of memory bandwidth ... I didn't know "terabytes (TB)" was a unit of memory bandwidth...
Bit of an embarrassing typo, they do later qualify it as 3.7TB/s
Re: Intel Gaudi 3 AI Accelerator
#6> the only MLPerf-benchmarked alternative for LLMs on the market
I hope to work on this for AMD MI300x soon. My company just got added to the MLCommons organization.
Re: Intel Gaudi 3 AI Accelerator
#7> Twenty-four 200 gigabit (Gb) Ethernet ports are integrated into every Intel Gaudi 3 accelerator
WHAT‽ It's basically got the equivalent of a 24-port, 200-gigabit switch built into it. How does that make sense? Can you imaging stringing 24 Cat 8 cables between servers in a single rack? Wait: How do you even decide where those cables go? Do you buy 24 Gaudi 3 accelerators and run cables directly between every single one of them so they can all talk 200-gigabit ethernet to each other?
Also: If you've got that many Cat 8 cables coming out the back of the thing how do you even access it? You'll have to unplug half of them (better keep track of which was connected to what port!) just to be able to grab the shell of the device in the rack. 24 ports is usually enough to take up the majority of horizontal space in the rack so maybe this thing requires a minimum of 2-4U just to use it? That would make more sense but not help in the density department.
I'm imagining a lot of orders for "a gradient" of colors of cables so the data center folks wiring the things can keep track of which cable is supposed to go where.
Re: Intel Gaudi 3 AI Accelerator
#8128GB in one chip seems important with the rise of sparse architectures like MoE. Hopefully these are competitive with Nvidia's offerings, though in the end they will be competing for the same fab space as Nvidia if I'm not mistaken.
AMD MI300x is 192GB.
Re: Intel Gaudi 3 AI Accelerator
#9Anyone have experience and suggestions for an AI accelerator?
Think prototype consumer product with total cost preferably < $500, definitely less than $1000.
Re: Intel Gaudi 3 AI Accelerator
#10> Twenty-four 200 gigabit (Gb) Ethernet ports are integrated into every Intel Gaudi 3 accelerator WHAT‽ It's basically got the equivalent of a 24-port, 200-gigabit switch built into it. How does that make sense? Can you imaging stringing 24 Cat 8 cables between servers in a single rack? Wait: How do you even decide where those cables go? Do you buy 24 Gaudi 3 accelerators and run cables directly between every single…
The amount of power that will use up is massive, they should've gone for some fiber instead