201–210 of 347 posts
I always thought that a real time scalable architecture would be beneficial. It's refreshing to see someone working on it, and exciting to see that it's nVidia. I always pictured a CPU with variable bit-width. Like a 256-bit ALU that could partition itself down into 16 or 32 bit ALU's as the workload allowed.
Earlier quoted context omitted.
Hypothetically, from an ISA perspective, why couldn't Intel and AMD extend x86-64 more fully with SIMD / MIMD instructions? (as in, way more fully than MMX / SSE / AVX) Naive question, because I literally don't know the link between CPU instruction stream and GPGPU instruction stream. But it seems like there would be an opportunity to seize the higher (open) ground at the ISA level, and then force Nvidia to implement…
That is what Intel tried to do with Larabee and failed spectacularly.
But in the late 80s and early 90s they actually had the i860 and its variants: https://en.wikipedia.org/wiki/Intel_i860
which contained a graphics processor! That wasn't a particular success but it landed us the MMX ISA on x86.
Surprisingly enough, Intel went then to make similar mistakes with Itanium, and then with Larabee. But both things landed us features on x86.
Earlier quoted context omitted.
That’s a fancy kitchen.
And he sure has a LOT of really nice spatulas! His Spatula City Frequent Buyer Card must have a lot of stamps on it. https://www.youtube.com/watch?v=2XbCWmY0eqY
Earlier quoted context omitted.
It would suffice for NVIDIA to open-source enough specifications and perhaps some subset of core software to enable others to build high quality open source (or even proprietary) software that targets NVIDIA's architecture. They can't hire every programmer in the world; if other programmers can build high-performance software that takes advantage of their platform, that increases the value of their hardware. Your com…
Anyone is free to target PTX and do their own compiler on top. In fact, given that it is there since version 3, there are compilers available for almost all major programing languages, including managed ones. While OpenCL is a C world, and almost no one cares about the C++ extensions and even less vendors care about SPIR-V. Also the community doesn't seem to be bothered that for a long time, the only SYCL implementat…
Earlier quoted context omitted.
'brain floating point' is also bad, but no one cares because it's just bfloat16. If this becomes popular, it will just be tf32 or tfloat32 or something.
I never knew what the “b” in “bfloat” was in all these new DL chips… until today. Man that’s bad.
Earlier quoted context omitted.
Pretty much everyone these days uses a library for driving the GPU calculations. And they tend to either support multiple hardware targets directly (TensorFlow) or have API-compatible replacements (CuPy/NumPy). So the lock-in risk here is that you might have to run your stuff on CPU if future NVIDIA GPUs are too overpriced. I mean they are super expensive. But there's nothing that comes close to their cuBLAS library…
Yeah I think most people don't quite appreciate the difficulty and cost of optimizing for hardware and continually maintaining that through hardware cycles. In keeping things closed source Nvidia products have both the advantages of being easier to on-board due to simpler abstraction, and faster technical progress because there is less pushback from myriad parties when big inconvenient changes might need to happen at…
Actually, I wonder why we went with un-compilable Java bytecode and JIT instead of advancing projects like gcj.
Earlier quoted context omitted.
It would suffice for NVIDIA to open-source enough specifications and perhaps some subset of core software to enable others to build high quality open source (or even proprietary) software that targets NVIDIA's architecture. They can't hire every programmer in the world; if other programmers can build high-performance software that takes advantage of their platform, that increases the value of their hardware. Your com…
Anyone is free to target PTX and do their own compiler on top. In fact, given that it is there since version 3, there are compilers available for almost all major programing languages, including managed ones. While OpenCL is a C world, and almost no one cares about the C++ extensions and even less vendors care about SPIR-V. Also the community doesn't seem to be bothered that for a long time, the only SYCL implementat…
Bothered has nothing to do with it. Implementing low level toolchains generally seems to require both a gargantuan effort and an incredible depth of knowledge. If it didn't, I think tooling and languages in general would be significantly better across the board.
What am I supposed to do, implement a SYCL compiler on my own? Forget it - I'll just keep writing GLSL compute shaders or OpenCL kernels until someone with lots of resources is able to foot the initial bill for a fully functional and open source implementation.
Earlier quoted context omitted.
Physics simulations. There's a rule of thumb that to get an n-bit accurate result after a long chain of calculations, intermediate results should be stored with 2n bits. Often using the full dynamic range of a float is necessary because the magnitude of different physical phenomena varies so wildly. I guess people do store intermediate results in floats in order to take advantage of GPU acceleration. However, once yo…
I've always argued that if you are getting close to having to worry about underflow, overflow etc. then you have an ill-conditioned problem and just increasing the size of your intermediate results won't help you a huge amount because you need more precision from your inputs. There are very few fields where you need more than the 7 decimal digits afforded by floats. Maybe the only exceptions are in astrophysics.
Earlier quoted context omitted.
The nvidia pgi compiler compiles CUDA to multi-core x86-64. There are other third-party compilers for CUDA->x86-64 (one LLVM-based one from Intel). There is a "library replacement" for CUDA from AMD called HIP, that you can use to map CUDA programs to ROCm. But... it doesn't work very well. NVIDIA also open-sourced CUDA support for Clang and LLVM. So anybody can extend clang to map CUDA to any hardware supported by L…
It sounds like people want nvidia to write drivers for AMD. This criticism makes even less sense when any bystander could implement CUDA suppport on AMD by connecting open source software.
You aren't seriously implying than any bystander is capable of extending LLVM to map CUDA to SPIR-V? What percentage of present day gainfully employed software engineers do you suppose even has the background knowledge? How many hours do you suppose the work would require?
Extending the tensor Ops to FP64 is an interesting, if not surprising, design choice. Are there many applications sure to leverage this capability? Aside from HPL, of course.
I am suspecting that this is specifically targeted at the HPC market as the lack of FP64 has always been a hindrance to HPC deployment. You have to remember that the HPC market is $35B today. HPE makes $3B a year alone from that, maybe more with Cray acquisition. So it's no surprise that NVIDIA wants to position themselves on that market. Plus, you have too look at the long game with MLX acquisition ( heavy player in…