Nvidia Ampere GA102 GPU Architecture [pdf] • Hacker News Board

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#21

Post by modeless » Sat, Sep 19, 2020, 3:09 AM UTC

apatheticonion wrote: post #16

01100011 wrote: post #3

Earlier quoted context omitted.

When you run up against the limits of Moore's law and the end of Dennard Scaling you have to get dirty and do what it takes to get more performance. Personally, I think CPU architecture became too complicated for my taste after the 68k. So what?

As someone with the most basic understanding of CPUs and assembly, why do you say that?

Advances in semiconductor fabrication are no longer increasing single-threaded performance of general purpose CPU code very much. It's still possible to increase performance by orders of magnitude, but it requires giving up flexibility by using dedicated hardware acceleration blocks such as tensor cores, plus going massively parallel with various combinations of SIMD and threading. That all makes software immensely more complex and less portable.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#22

Post by modeless » Sat, Sep 19, 2020, 3:13 AM UTC

bronxbomber92 wrote: post #4

amelius wrote: post #2

These GPU architectures are too complicated for my taste. It reminds me of the days of segmented memory, and makes me feel like having to jump through hoops all the time. I'm curious if people have tried to invent something more elegant, and where these approaches have failed. Also, the fact that NVidia has probably patented this particular architecture makes it less interesting for me to really dig into.

There is roughly equal levels of complexity in any modern, high performance, general purpose/programmable chip.

But it's not equally exposed to the programmer. The branch predictor in an Intel CPU may be insanely complex but when writing code you usually don't need to care. On the other hand if you want to use a GPU you have to care about a huge amount of complexity right from the start.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#23

Post by lajawfe » Sat, Sep 19, 2020, 3:35 AM UTC

Can the FP32 and Tensor core modules compute at the same time, or are they an abstraction over the underlying silicon, and we can only utilize one at a time?

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#24

Post by _kbh_ » Sat, Sep 19, 2020, 3:42 AM UTC

lajawfe wrote: post #23

Can the FP32 and Tensor core modules compute at the same time, or are they an abstraction over the underlying silicon, and we can only utilize one at a time?

Pages 16 and 17 in the PDF have graphs that show them being scheduled concurrently.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#25

Post by _kbh_ » Sat, Sep 19, 2020, 4:42 AM UTC

PaulKeeble wrote: post #17

jagger27 wrote: post #13

Earlier quoted context omitted.

> The FP32 doubling, is one of the most important bits here. But fortunately for programmers, this doesn't really change how you do your code. Early benchmarks are showing games under-performing quite a bit in the worst cases. The crux of the issue is that it's not /exactly/ a no-compromise doubling of FP32. Each data path per SM can either do 2xFP32 or 1xINT32/1xFP32 per clock cycle. So if your game or application h…

Its not uncommon for GPU workloads in games to max out about 20% INT32 calculations, but alas its enough to drop the FP32 performance quite a bit. I suspect Nvidia next time will probably separate out the INT32 and 2x FP32 units and gradually move towards going towards a better ratio of hardware that better suits the usual workload split.

Due to the lower amount of INT32 in game loads as you stated, I don't think that separating INT32 and FP32 hardware makes a lot of sense, because you can share a substantial amount of the hardware between the two overall leading to space savings.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#26

Post by kllrnohj » Sat, Sep 19, 2020, 4:58 AM UTC

jagger27 wrote: post #13

dragontamer wrote: post #5

I posted this a day or two ago: The A100 whitepaper "spoiled" a lot of these factoids already. ( https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent ...) The new bit seems to be the doubling of FP32 "CUDA cores" (I really hate that word: when Intel or AMD double their CPU pipelines it doesn't mean that they're selling more cores, it means their cores got wider... anyway). A100 didn't have this feature (I ass…

> The FP32 doubling, is one of the most important bits here. But fortunately for programmers, this doesn't really change how you do your code. Early benchmarks are showing games under-performing quite a bit in the worst cases. The crux of the issue is that it's not /exactly/ a no-compromise doubling of FP32. Each data path per SM can either do 2xFP32 or 1xINT32/1xFP32 per clock cycle. So if your game or application h…

> though you get the benefit of parallel INT32 execution

Parallel INT32 was added with the last generation, in Turing. See page 13 of https://www.nvidia.com/content/dam/en-zz/Solutions/design-vi...

So nvidia split out the INT32 from FP32 last gen to make them independent, then re-added FP32 to the INT32 but kept it as 2 datapaths.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#27

Post by kllrnohj » Sat, Sep 19, 2020, 5:07 AM UTC

modeless wrote: post #22

bronxbomber92 wrote: post #4

Earlier quoted context omitted.

There is roughly equal levels of complexity in any modern, high performance, general purpose/programmable chip.

But it's not equally exposed to the programmer. The branch predictor in an Intel CPU may be insanely complex but when writing code you usually don't need to care. On the other hand if you want to use a GPU you have to care about a huge amount of complexity right from the start.

Except you really do need to care about this complexity on CPUs. Things like cache locality & predictable access patterns are critical to achieving good CPU performance. This is why there's things like data-oriented design, SoA vs. AoS, and Z-order curves. It's also why linked-lists are so incredibly awful in practice, despite having superb algorithmic performance in theory.

A big reason programming for CPUs doesn't seem as complex is because the vast, vast majority of time nobody actually cares about CPU performance. We all just prefer to pretend a runtime or JIT or compiler managed to magically make a language that's god-awful horrendous on modern CPUs run fast. They didn't, we just all look the other way though.

The difference between CPUs & GPUs is when people reach for GPUs, such as for games or HPC, those are also the people that care a lot about performance. And guides like this are for them.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#28

Post by kllrnohj » Sat, Sep 19, 2020, 5:20 AM UTC

modeless wrote: post #6

amelius wrote: post #2

These GPU architectures are too complicated for my taste. It reminds me of the days of segmented memory, and makes me feel like having to jump through hoops all the time. I'm curious if people have tried to invent something more elegant, and where these approaches have failed. Also, the fact that NVidia has probably patented this particular architecture makes it less interesting for me to really dig into.

It's a good question actually. Intel tried to make a GPU called Larrabee that was mostly a bunch of small x86 cores with giant vector units. Turns out that it couldn't compete in rendering performance on existing games (in 2010) without the fixed function units that GPUs have, so they canceled it as a GPU. It did result in the AVX-512 instruction set though. I think the idea still has promise but there's a chicken an…

Although I don't think the programming model with Larrabee would have really been any simpler. You still face many of the same issues that you do with GPUs, although being SIMD instead of SIMT would actually make it slightly harder to work with.

The actual hard part with GPUs is ensuring you can divide up the work and that it doesn't branch within a given chunk size. You have those same issues when trying to leverage a many-core CPU with AVX-512. You still want to keep those AVX-512 units loaded, which means work units of 16 FP32's must all take the same "branch" - not really any different from feeding warps on a GPU. And you've still got to scale across dozens if not hundreds of CPU cores.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#29

Post by modeless » Sat, Sep 19, 2020, 6:23 AM UTC

kllrnohj wrote: post #28

modeless wrote: post #6

Earlier quoted context omitted.

It's a good question actually. Intel tried to make a GPU called Larrabee that was mostly a bunch of small x86 cores with giant vector units. Turns out that it couldn't compete in rendering performance on existing games (in 2010) without the fixed function units that GPUs have, so they canceled it as a GPU. It did result in the AVX-512 instruction set though. I think the idea still has promise but there's a chicken an…

Although I don't think the programming model with Larrabee would have really been any simpler. You still face many of the same issues that you do with GPUs, although being SIMD instead of SIMT would actually make it slightly harder to work with. The actual hard part with GPUs is ensuring you can divide up the work and that it doesn't branch within a given chunk size. You have those same issues when trying to leverage…

AVX-512 has an execution mask and usually should be programmed with an SIMT-like model (e.g. with ispc). Writing SIMT kernels or chunking up the work is not the hard part.

The actual actual hard part with GPUs is writing portable code in the face of a million edge cases due to different proprietary hardware architectures and buggy drivers, which you can't test without actually buying and maintaining whole rooms full of hardware. Reducing fixed function parts of the hardware and using a documented ISA, as Larrabee tried, would help with that.

Re: Nvidia Ampere GA102 GPU Architecture [pdf]

#30

Post by modeless » Sat, Sep 19, 2020, 6:56 AM UTC

kllrnohj wrote: post #27

modeless wrote: post #22

Earlier quoted context omitted.

But it's not equally exposed to the programmer. The branch predictor in an Intel CPU may be insanely complex but when writing code you usually don't need to care. On the other hand if you want to use a GPU you have to care about a huge amount of complexity right from the start.

Except you really do need to care about this complexity on CPUs. Things like cache locality & predictable access patterns are critical to achieving good CPU performance. This is why there's things like data-oriented design, SoA vs. AoS, and Z-order curves. It's also why linked-lists are so incredibly awful in practice, despite having superb algorithmic performance in theory. A big reason programming for CPUs doesn't…

I don't think we're fundamentally in disagreement. But I will say that there's a huge amount of value in CPUs not forcing you to care about the complexity when it doesn't matter, and it doesn't always matter.