Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance per dollar is getting faster and cheaper

wafer.ai

Post reply on HN
Print view View original

21–30 of 151 posts

Re: Performance per dollar is getting faster and cheaper

#21

Post by kingstnap »

minraws wrote: post #2

Can you folks add performance per watt as a metric to these comparisons, I honestly want to understand where AMD fits in the stack in terms of actual performance to dollars. I have had talks with companies wanting to build data centers outside of US and find it hard to source anything Nvidia in sufficient capacity and scale. If AMD is competitive performance per watt and roughly reliable in terms of software support…

A DGX B200 costs like ~$0.5 M and uses around 14 kW.

If you plan to run it straight for 8 years 100% max usage thats around 1 GWhr.

A gigawatt hour is a lot of energy but its not that much compared to the price of the actual machine. In Germany for example with its expensive energy thats about €100k worth, which spread over 8 years is pretty minor compared to the up front half mill.

The real issue with high power consumption is not really the cost of energy but the limited powersupply you can get for a datacenter. A more efficient setup is highly desirable because it means you can fit more in the limited power hookup.

Re: Performance per dollar is getting faster and cheaper

#22

Post by nullc »

Schiendelman wrote: post #18

I'm not surprised to see competition with Blackwell. Rubin is 5x faster than Blackwell at inference - Blackwell is the last generation Nvidia didn't optimize specifically for inference. If I'm missing something, please let me know!

how do you get 5x faster at inference when inference is memory bandwidth limited? getting 5x the memory bandwidth of a h100 seems physically difficult.

Re: Performance per dollar is getting faster and cheaper

#23

Post by Schiendelman »

nullc wrote: post #22
Schiendelman wrote: post #18

I'm not surprised to see competition with Blackwell. Rubin is 5x faster than Blackwell at inference - Blackwell is the last generation Nvidia didn't optimize specifically for inference. If I'm missing something, please let me know!

how do you get 5x faster at inference when inference is memory bandwidth limited? getting 5x the memory bandwidth of a h100 seems physically difficult.

Rubin has 22TB/s of memory bandwidth vs Blackwell's 8TB/s. NVLink 6 doubles interconnect speed. Plus they're moving to 3nm from ~4nm.

(Previously this comment said Rubin did native NVFP4, but Blackwell does too! Rubin just also trains with native NVFP4, which Blackwell does not.)

Re: Performance per dollar is getting faster and cheaper

#24

Post by latchkey »

minraws wrote: post #2

Can you folks add performance per watt as a metric to these comparisons, I honestly want to understand where AMD fits in the stack in terms of actual performance to dollars. I have had talks with companies wanting to build data centers outside of US and find it hard to source anything Nvidia in sufficient capacity and scale. If AMD is competitive performance per watt and roughly reliable in terms of software support…

> I have never seen a company use AMD outside of wafer and a couple others mostly in US.

Just because you haven't seen it doesn't mean it doesn't exist.

We've serviced over 700 customers on our MI300x.

Re: Performance per dollar is getting faster and cheaper

#27

Post by boredatoms »

Schiendelman wrote: post #23
nullc wrote: post #22

Earlier quoted context omitted.

how do you get 5x faster at inference when inference is memory bandwidth limited? getting 5x the memory bandwidth of a h100 seems physically difficult.

Rubin has 22TB/s of memory bandwidth vs Blackwell's 8TB/s. NVLink 6 doubles interconnect speed. Plus they're moving to 3nm from ~4nm. (Previously this comment said Rubin did native NVFP4, but Blackwell does too! Rubin just also trains with native NVFP4, which Blackwell does not.)

Moving to lower bits is not a slam dunk, the model itself might degrade too much

Re: Performance per dollar is getting faster and cheaper

#28

Post by zackangelo »

Schiendelman wrote: post #23
nullc wrote: post #22

Earlier quoted context omitted.

how do you get 5x faster at inference when inference is memory bandwidth limited? getting 5x the memory bandwidth of a h100 seems physically difficult.

Rubin has 22TB/s of memory bandwidth vs Blackwell's 8TB/s. NVLink 6 doubles interconnect speed. Plus they're moving to 3nm from ~4nm. (Previously this comment said Rubin did native NVFP4, but Blackwell does too! Rubin just also trains with native NVFP4, which Blackwell does not.)

Blackwell supports nvfp4 natively.

Re: Performance per dollar is getting faster and cheaper

#30

Post by Schiendelman »

boredatoms wrote: post #27
Schiendelman wrote: post #23

Earlier quoted context omitted.

Rubin has 22TB/s of memory bandwidth vs Blackwell's 8TB/s. NVLink 6 doubles interconnect speed. Plus they're moving to 3nm from ~4nm. (Previously this comment said Rubin did native NVFP4, but Blackwell does too! Rubin just also trains with native NVFP4, which Blackwell does not.)

Moving to lower bits is not a slam dunk, the model itself might degrade too much

Of course, but for most workflows it's fine.
Post reply on HN