I always thought that a real time scalable architecture would be beneficial. It's refreshing to see someone working on it, and exciting to see that it's nVidia. I always pictured a CPU with variable bit-width. Like a 256-bit ALU that could partition itself down into 16 or 32 bit ALU's as the workload allowed.
Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
91–100 of 347 posts
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#92Earlier quoted context omitted.
Hypothetically, from an ISA perspective, why couldn't Intel and AMD extend x86-64 more fully with SIMD / MIMD instructions? (as in, way more fully than MMX / SSE / AVX) Naive question, because I literally don't know the link between CPU instruction stream and GPGPU instruction stream. But it seems like there would be an opportunity to seize the higher (open) ground at the ISA level, and then force Nvidia to implement…
That is what Intel tried to do with Larabee and failed spectacularly.
I wouldn't say they failed. I'd say they gave up on it before product maturity.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#93Earlier quoted context omitted.
The large majority of researchers and business getting into NVidia products doesn't seem to find it that relevant, rather what tools, GPU programming languages and hardware they are able to put their hands on.
It's irrelevant to researchers. Research operates on rapid cycles: prototype, publish, move on. It does impact businesses. It doesn't prevent adoption for e.g. deep learning, but I haven't seen e.g. GPU-based databases reach broad adoption, or many other places where MIMD/SIMD would reduce costs or improve performance. Using classical hardware is clearly cheaper than the business risk and engineering time of relying…
NVidia is not to blame if the competition is stuck using C, printf debugging for computing shaders, cannot make their minds about which bytecode to support for heterogenous GPGPU programming.
The situation is so bad that OpenCL 1.2 got promoted to OpenCL 3.0 and SYSCL is now backend independent, while hip only works on Linux.
As for Python, guess who is on the forefront of GPU Programming with Python,
https://www.nvidia.cn/gtc/session-catalog/?search=python
41 results, including CUDA based JIT improvements.
Meanwhile, at IWOCL & SYCLcon 2020,
https://www.iwocl.org/iwocl-2020/conference-program
2 sessions, where it is mentioned that PyFR might need OpenCL 3.0 extensions going forward.
So if competition is not able to provide, most just get to buy NVidia.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#94Probably even more closed than ever. They tend to become more and more restrictive with every new hardware generation. I wonder where their promised open source announcement they preannounced before.
Yeah. That's been my general problem with adopting NVidia for anything. They make good hardware, but there's a lot of lock-in, and not a lot of transparency. That introduces business risk. I'm not in a position where I need GPGPU, but if there wasn't that risk, and generally there were mature, open standards, I'd definitely use it. The major breakpoint would be when libraries like Numpy do it natively, and better yet…
> They make good hardware, but there's a lot of lock-in, and not a lot of transparency.
This sounds like you'd like NVIDIA to open-source all their software. I see this type of request a lot, but I don't see it happening.
NVIDIA's main competitive advantage over AMD and Intel is its software stack. AMD could release a 2x powerful GPGPU tomorrow for half the price and most current NVIDIA users wouldn't care because what good is that if you can't program it? AMD software offer is just poor, of course they open-source everything, they don't make any software worth buying.
ARM and Intel make great software (the Intel MKL, Intel SVML, ... libraries, icc, ifort, ... compiler), and it doesn't open-source any of that either for the same reasons as NVIDIA.
Intel and NVIDIA employ a lot of people to develop their software stacks. These people aren't probably very cheap. AMD strategy is to save a lot of money in software development, maybe hoping that the open-source communities or Intel and NVIDIA will do it for free.
I also see these requests that Intel and NVIDIA should open-source everything together with the explanation that "I need this because I want to buy AMD stuff". That, right there, is the reason why they don't do it.
You want to know why NVIDIA has 99% of the Cloud GPGPU hardware market and AMD 1%? If you think 10.000$ for a V100 is expensive, do the math on how much does an AMD MI50 costs: 5000$ for the hardware, and then a team of X >100k$ engineers (how much do you think AI GPGPU engineers cost?) working for N years just to play catch on the part of the software stack that NVIDIA gives you with a V100 for free. That goes into multiple million dollars more expensive really quickly.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#95Earlier quoted context omitted.
Yeah. That's been my general problem with adopting NVidia for anything. They make good hardware, but there's a lot of lock-in, and not a lot of transparency. That introduces business risk. I'm not in a position where I need GPGPU, but if there wasn't that risk, and generally there were mature, open standards, I'd definitely use it. The major breakpoint would be when libraries like Numpy do it natively, and better yet…
> Intel keeps getting it right with numerical libraries. They're open. They work well. They work on AMD. What Intel numerical libraries are you thinking of? When I think of Intel numerical libraries, the first that comes to mind is MKL. MKL is neither open-source nor does it work well on AMD without some fragile hacks [0]. [0] https://www.pugetsystems.com/labs/hpc/How-To-Use-MKL-with-AM...
Well, OP didn't say MKL works well on AMD. But you can at least run it on a non-Intel CPU. Compare CUDA.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#96For those in the industry: When a new generation like this is released, will a typical AI company replace the current GPUs? Is there a chance to acquire the older versions for private use or is it too early for that?
A fun quick of GPU pricing economics is that on Google Cloud platform, the relatively recent T4 (on Turing and has FP16 support) is cheaper than the ancient K80s. https://cloud.google.com/compute/gpus-pricing
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#97The TensorFloat-32 results look really impressive, but yikes that is not a good name. "TensorFloat" is extremely confusable with "TensorFlow," and it would really more accurately be called a 19 bit format.
'brain floating point' is also bad, but no one cares because it's just bfloat16. If this becomes popular, it will just be tf32 or tfloat32 or something.
I never knew what the “b” in “bfloat” was in all these new DL chips… until today. Man that’s bad.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#98The numbers for their SATURNV supercomputer are either untrue or absolutely staggering. 4.6 exaflops? #1 on the Top 500 list of supercomputers just barely passed 200 petaflops at peak performance. If you add up the entire list you only get 1.65 exaflops. And LINPACK isn't usually network-bound. How can this possibly be true?
They're counting TF32, which is a 19 bit format and comparing it to FP64.
It's tensor Ops, and so the precision might be even lower than single. It could be BF16, for example.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#99Earlier quoted context omitted.
The large majority of researchers and business getting into NVidia products doesn't seem to find it that relevant, rather what tools, GPU programming languages and hardware they are able to put their hands on.
It's irrelevant to researchers. Research operates on rapid cycles: prototype, publish, move on. It does impact businesses. It doesn't prevent adoption for e.g. deep learning, but I haven't seen e.g. GPU-based databases reach broad adoption, or many other places where MIMD/SIMD would reduce costs or improve performance. Using classical hardware is clearly cheaper than the business risk and engineering time of relying…
There's definitely a chicken-and-egg problem of the current user and developer base of GPGPU apps being a very tolerant bunch. "It's just a flesh wound" they say about a lot of things that are prohibitive to normal app developers. Maybe it's the natural order of things, or maybe the the "incompatible proprietary C++ dialects with different kinds of crashy drivers on each OS" approach will be suffiently unpalatable to some future generation of programmers.
Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU
#100They show a fictional render of the main chip surrounded by four large gold coated leadless packages. What are those supposed to be?
Probably a heatsink