Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

nicolasdickenmann.com

Post reply on HN
Print view View original

81–90 of 93 posts

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#81

Post by adrian_b »

tverbeure wrote: post #71
adrian_b wrote: post #48

Earlier quoted context omitted.

They were also full of lies when they have started their GPGPU adventure (like also today). For a few years they have repeated continuously how GPGPU can provide about 100 times more speed than CPUs. This has always been false. GPUs are really much faster, but their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs. This is impressiv…

> GPGPU can provide about 100 times more speed than CPUs Ok. You're talking about performance. > their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs Now you're talking about perf/W. > This is impressive, but very far from the "100" factor originally claimed by NVIDIA. That's because you're comparing apples to apples per apple car…

For determining the maximum performance achievable, the performance per watt is what matters, as the power consumption will always be limited by cooling and by the available power supply.

Even if we interpret the NVIDIA claim as referring to the performance available in a desktop, the GPU cards had power consumptions at most double in comparison with CPUs. Even with this extra factor there has been more than an order of magnitude between reality and the NVIDIA claims.

Moreover I am not sure whether around 2010 and before that, when these NVIDIA claims were frequent, the power permissible for PCIe cards had already reached 300 W, or it was still lower.

In any case the "100" factor claimed by NVIDIA was supported by flawed benchmarks, which compared an optimized parallel CUDA implementation of some algorithm with a naive sequential implementation on the CPU, instead of comparing it with an optimized multithreaded SIMD implementation on that CPU.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#82

Post by fooblaster »

gdiamos wrote: post #28
fooblaster wrote: post #24

Earlier quoted context omitted.

It was definitely luck, greg. And Nvidia didn't invent deep learning, deep learning found nvidias investment in CUDA.

I remember it differently. CUDA was built with the intention of finding/enabling something like deep learning. I thought it was unrealistic too and took it on faith in people more experienced than me, until I saw deep learning work. Some of the near misses I remember included bitcoin. Many of the other attempts didn't ever see the light of day. Luck in english often means success by chance rather than one's own effor…

CUDA was profitable very early because of oil and gas code, like reverse time migration and the like. There was no act of incredible foresight from jensen. In fact, I recall him threatening to kill the program if large projects that made it not profitable failed, like the Titan super computer at oak ridge.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#83

Post by tverbeure »

adrian_b wrote: post #81
tverbeure wrote: post #71

Earlier quoted context omitted.

> GPGPU can provide about 100 times more speed than CPUs Ok. You're talking about performance. > their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs Now you're talking about perf/W. > This is impressive, but very far from the "100" factor originally claimed by NVIDIA. That's because you're comparing apples to apples per apple car…

For determining the maximum performance achievable, the performance per watt is what matters, as the power consumption will always be limited by cooling and by the available power supply. Even if we interpret the NVIDIA claim as referring to the performance available in a desktop, the GPU cards had power consumptions at most double in comparison with CPUs. Even with this extra factor there has been more than an order…

At the time, desktop power consumption was never a true limiter. Even for the notorious GTX 480, TDP was only 250 W.

That aside, it still didn't make sense to compare apples to apples per apple cart...

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#84

Post by 0x457 »

imtringued wrote: post #30
oivey wrote: post #20

Earlier quoted context omitted.

Even that is arguably not lucky, it just followed a non-obvious trajectory. Graphics uses a fair amount of linear algebra, so people with large scale physical modeling needs (among many) became interested. To an extent the deep learning craze kicked off because of developments in computation on GPUs enabled economical training.

Nvidia started their GPGPU adventure by acquiring a physics engine and porting it over to run on their GPUs. Supporting linear algebra operations was pretty much the goal from the start.

That physics engine is an example of a dead-end.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#85

Post by 0x457 »

adrian_b wrote: post #48
imtringued wrote: post #30

Earlier quoted context omitted.

Nvidia started their GPGPU adventure by acquiring a physics engine and porting it over to run on their GPUs. Supporting linear algebra operations was pretty much the goal from the start.

They were also full of lies when they have started their GPGPU adventure (like also today). For a few years they have repeated continuously how GPGPU can provide about 100 times more speed than CPUs. This has always been false. GPUs are really much faster, but their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs. This is impressiv…

> A few years later, AMD has followed the NVIDIA example.

When bitcoin was still profitable to mine on GPUs, AMD's performed better due to not being segmented like NVIDIA cards. It didn't help AMD, not that it matters. AMD started segmenting because they couldn't make a competitive card at a competitive price for the consumer market.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#86

Post by 0x457 »

pjc50 wrote: post #66
wtallis wrote: post #2

It's amazing to step back and look at how much of NVIDIA's success has come from unforeseen directions. For their original purpose of making graphics chips, the consumer vs pro divide was all about CAD support and optional OpenGL features that games didn't use. Programmable shaders were added for the sake of graphics rendering needs, but ended up spawning the whole GPGPU concept, which NVIDIA reacted to very well wit…

The counter question is: why have AMD been so bad by comparison?

Because AMD only every had one strategy for software:

Look at it as, we're David against the Goliath, we're going to utilize open-source because that's how you win.

Then make zero effecting trying to increase adoption, which then result in reducing support for it and eventually giving up.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#87

Post by [deleted] »

philipkglass wrote: post #78
jjmarr wrote: post #3

FP64 performance is limited on consumer because the US government deems it important to nuclear weapons research. Past a certain threshold of FP64 throughput, your chip goes in a separate category and is subject to more regulation about who you can sell to and know-your-customer. FP32 does not matter for this threshold. https://en.wikipedia.org/wiki/Adjusted_Peak_Performance It is not a market segmentation tactic and…

It's surprising that this restriction continues to linger at all. The newest nuclear warhead models in the US arsenal were developed in the 1970s, when supercomputer performance was well below 1 gigaflop. When the US stopped testing nuclear warheads in 1992, top end supercomputers were under 10 gigaflops. The only thing the US arsenal needs faster computers for is simulating the behavior of its aging warhead stockpil…

[deleted]

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#88

Post by gdiamos »

fooblaster wrote: post #82
gdiamos wrote: post #28

Earlier quoted context omitted.

I remember it differently. CUDA was built with the intention of finding/enabling something like deep learning. I thought it was unrealistic too and took it on faith in people more experienced than me, until I saw deep learning work. Some of the near misses I remember included bitcoin. Many of the other attempts didn't ever see the light of day. Luck in english often means success by chance rather than one's own effor…

CUDA was profitable very early because of oil and gas code, like reverse time migration and the like. There was no act of incredible foresight from jensen. In fact, I recall him threatening to kill the program if large projects that made it not profitable failed, like the Titan super computer at oak ridge.

I remember it being less profitable than graphics for a long time.

It did make money that would be interesting to a startup, but not to a public company.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#89

Post by h3lp »

tverbeure wrote: post #71
adrian_b wrote: post #48

Earlier quoted context omitted.

They were also full of lies when they have started their GPGPU adventure (like also today). For a few years they have repeated continuously how GPGPU can provide about 100 times more speed than CPUs. This has always been false. GPUs are really much faster, but their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs. This is impressiv…

> GPGPU can provide about 100 times more speed than CPUs Ok. You're talking about performance. > their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs Now you're talking about perf/W. > This is impressive, but very far from the "100" factor originally claimed by NVIDIA. That's because you're comparing apples to apples per apple car…

Well, power envelope IS the limit in many applications; anyone can build a LOBOS (Lots Of Boxes On Shelves) supercomputer, but data bandwidth and power will limit its usefullness and size. Everyone has a power budget. For me, it's my desk outlet capacity (1.5kW); for a hyperscaler, it's the capacity of the power plant that feeds their datacenter (1.5GW); we both cannot exceed Pmax * MIPS/W of computation.

Re: 15 years of FP64 segmentation, and why the Blackwell Ultra breaks the pattern

#90

Post by tverbeure »

h3lp wrote: post #89
tverbeure wrote: post #71

Earlier quoted context omitted.

> GPGPU can provide about 100 times more speed than CPUs Ok. You're talking about performance. > their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs Now you're talking about perf/W. > This is impressive, but very far from the "100" factor originally claimed by NVIDIA. That's because you're comparing apples to apples per apple car…

Well, power envelope IS the limit in many applications; anyone can build a LOBOS (Lots Of Boxes On Shelves) supercomputer, but data bandwidth and power will limit its usefullness and size. Everyone has a power budget. For me, it's my desk outlet capacity (1.5kW); for a hyperscaler, it's the capacity of the power plant that feeds their datacenter (1.5GW); we both cannot exceed Pmax * MIPS/W of computation.

All of that may be true but it’s irrelevant.

If you’re dividing perf by perf/W, it makes no sense to yell “it’s not equal to 100!” You simply failed at dimension analysis taught in high school.

Post reply on HN