31–40 of 114 posts
This tweet appears to be taking the original material out of context to misrepresent it: > Rewrite the attention kernel to be persistent. This gives better performance at low-contexts. However, fp16 at large context has suffered a bit due to a ptxas instruction scheduling issue in the softmax partition. fp8 is ~100 tflops faster when the kernel name has "cutlass" in it. The charitable reading is that, on certain kern…
This tweet appears to be taking the original material out of context to misrepresent it: > Rewrite the attention kernel to be persistent. This gives better performance at low-contexts. However, fp16 at large context has suffered a bit due to a ptxas instruction scheduling issue in the softmax partition. fp8 is ~100 tflops faster when the kernel name has "cutlass" in it. The charitable reading is that, on certain kern…
# Up to 150 TFLOPS faster for fp8!
if specialization.constants["dtype"] == gl.float8e5:
name = "cutlass_" + nameThis tweet appears to be taking the original material out of context to misrepresent it: > Rewrite the attention kernel to be persistent. This gives better performance at low-contexts. However, fp16 at large context has suffered a bit due to a ptxas instruction scheduling issue in the softmax partition. fp8 is ~100 tflops faster when the kernel name has "cutlass" in it. The charitable reading is that, on certain kern…
It's literally in the code.
So, what is Cutlass, can someone explain whether checking for kernel names makes sense here or is a form of cheating? https://docs.nvidia.com/cutlass/index.html
In `libnvidia-nvvm.so` the string `cutlass` appears right after `Memory Dependence Analysis` and `memdep`. Perhaps it acts as an optimization attribute of some sort, where the compiler is allowed to make assumptions about the kernel's behavior that are not valid in general?
It is also part of the benchmarks game they play against each other.
Let's hope for Nvidia this is an innocent optimization only valid for internal kernels that cannot be applied in general.
So, what is Cutlass, can someone explain whether checking for kernel names makes sense here or is a form of cheating? https://docs.nvidia.com/cutlass/index.html
That's strange because the cutlass docs explicitly does NOT mention fp8 support. So it looks like it can be used nevertheless with fp8 by using the name hack.
I have small experience with compilers and llvm but youd be shocked how many things rely on names and parsing names If you have hundreds of passes that are complex and rely on various "contracts" like type names or some shit, then really crazy things like this can happen unintentionally and not maliciously
Web-developers are well aware of this too. Sincerely, Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0