21–30 of 221 posts
Note: this is old work, and much of the team working on TVM, and MLC were from OctoAI and we have all recently joined NVIDIA.
Earlier quoted context omitted.
It’s not terribly hard to port ML inference to alternative GPU APIs. I did it for D3D11 and the performance is pretty good too: https://github.com/Const-me/Cgml The only catch is, for some reason developers of ML libraries like PyTorch aren’t interested in open GPU APIs like D3D or Vulkan. Instead, they focus on proprietary ones i.e. CUDA and to lesser extent ROCm. I don’t know why that is. D3D-based videogames are h…
Isn't part of it because the first-party libraries like cuDNN are only available through CUDA? Nvidia has poured a ton of effort into tuning those libraries so it's hard to justify not using them.
OTOH, general-purpose compute instead of fixed-function blocks used by cuDNN enables custom compression algorithms for these weights which does help, by saving memory bandwidth. For example, I did custom 5 bits/weight quantization which works on all GPUs, no hardware support necessary, just simple HLSL codes: https://github.com/Const-me/Cgml?tab=readme-ov-file#bcml1-co...
Note: this is old work, and much of the team working on TVM, and MLC were from OctoAI and we have all recently joined NVIDIA.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation.
The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.
Note: this is old work, and much of the team working on TVM, and MLC were from OctoAI and we have all recently joined NVIDIA.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
Making AMD GPUs competitive for LLM inference https://news.ycombinator.com/item?id=37066522 (August 9, 2023 — 354 points, 132 comments)
Note: this is old work, and much of the team working on TVM, and MLC were from OctoAI and we have all recently joined NVIDIA.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
And that's what's happening, slowly anyway. Google, Apple and Amazon all have their own AI chips, Intel has Gaudi, AMD had their thing, and the software is at least working on more than just Nvidia. Which is a win. Even if it's not perfect. I'm personally hoping that everyone piles in on a standard like SYCL.
Earlier quoted context omitted.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
Not really. AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation. The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.
Earlier quoted context omitted.
Well sure, but in other GPU tasks, like Raytracing, the difference between these GPUs is far more pronounced. And AMD has passable Raytracing units (NVidias are better but the difference is bigger than these LLM results). If RAM is the main bottleneck then CPUs should be on the table.
CPUs have pitiful RAM bandwidth compared to GPUs. The speeds aren't so different but GPU RAM busses are wiiiiiiiide.
1) Compute Express Link (CXL):
https://en.wikipedia.org/wiki/Compute_Express_Link
PCIe vs. CXL for Memory and Storage:
Earlier quoted context omitted.
Not really. AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation. The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.
And I'm supposed to believe that HN is this amazing platform for technology and science discussions, totally unlike its peers...
Also I'd take HN as being being an amazing platform for the overall consistency and quality of moderation. Anything beyond that depends more on who you're talking to than where at.