Live data from Hacker News

Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

github.com

111–120 of 176 posts

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#111
post #68

Earlier quoted context omitted.

Thank you for explaining. I was so confused at how AMD was improving Quake performance with duck-like monikers.

Well, if it _looks_ like a high-performance texture renderer, and it _walks_ like a high-performance texture renderer...

If it looks like a benchmark and it quacks like a benchmark… duck?

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#112
post #97

Earlier quoted context omitted.

Why wouldn’t you just use inline assembly in that case?

There is no way to write inline SASS (assembly equivalent) for CUDA code. You can inline PTX, but PTX is a high level bytecode designed to be portable.

PTX is sometimes referred to as assembly, and it is an ISA, much lower level than C++. When people talk about writing inline assembly for CUDA, they mean PTX, and the C++ compiler’s inline assembly “asm” statement assumes PTX. For the most part you have much more control and ability to produce exactly the SASS you want when using PTX compared to when using C/C++.

https://developer.nvidia.com/blog/understanding-ptx-the-asse...

https://eunomia.dev/others/cuda-tutorial/02-ptx-assembly/

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#113
post #22

Someone really needs to learn to use `git commit --amend`. Almost 100 commits with pointless commit messages like "wip" or "x"? Be kinder to your reviewers...

Why do you care? Small commits are great for git bisect, and having to come up with a fancy message can break your flow. Code reviewers generally review a whole PR diff, not the individual commits. Fussing about commit messages smacks of prioritising aesthetics over functionality.

Do you frequently dig in to new codebases? I do, and commits that contain a functional, complete idea with a descriptive commit message are immensely useful to me for understanding why the code is the way it is.

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#114
post #17

Heh. Does anyone remember when almost 25 years ago ATI (AMD) caught manipulating the Quake III benchmarks by renaming the executables to ‘quack’? https://web.archive.org/web/20230929180112/https://techrepor... https://web.archive.org/web/20011108190056/https://hardocp.c... https://web.archive.org/web/20011118183932/www.3dcenter.de/a...

Every vendor does this to this day - and its a morally grey practice, drivers hijack and modify the rendering loops of popular games, fixing bugs, replacing shaders with more optimized versions, enabling faster codepaths in the driver etc. These changes are supposed to have minimal to no impact on the actual output, but sometimes vendors are really aggressive, and significantly degrade the outputs so that the game ca…

Sadly it's built into the vulkan protocol. Even a fully userspace driver arrangement with a microkernel ends up giving the driver access to the client's information. Of course it's forgeable the way it's done though so you could opt out if you really wanted to.

[1]: https://github.com/KhronosGroup/Vulkan-Headers/blob/main/inc...

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#115
post #17

Heh. Does anyone remember when almost 25 years ago ATI (AMD) caught manipulating the Quake III benchmarks by renaming the executables to ‘quack’? https://web.archive.org/web/20230929180112/https://techrepor... https://web.archive.org/web/20011108190056/https://hardocp.c... https://web.archive.org/web/20011118183932/www.3dcenter.de/a...

Or Intel checking for "GenuineIntel" in ICC's output: https://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Support...

Or Win 3.1 looking for whatever shibboleth was in MS-DOS and popping up a scary-looking message if it found another DOS? https://en.wikipedia.org/wiki/AARD_code

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#116
post #3

https://github.com/triton-lang/triton/pull/7298#discussion_r... > By disassembly of ptxas, it is indeed hard-coded that they have logic like: strstr(kernel_name, "cutlass"). > it is likely that, this is an unstable, experimental, aggressive optimization by NVIDIA, and blindly always enabling it may produce some elusive bugs.

Often not elusive bugs, but elusive performance. GPU compilers are hard: Once you've done the basics, trying to do further transforms in a mature compiler will almost always produced mixed results. Some kernels will go faster, some will go slower, and you're hoping to move the balance and not hit any critical kernel too hard in your efforts to make another go faster.

An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup.

My experience is with non-Nvidia GPU systems, but this feels like a familiar situation. They probably found something that has great outcomes for one set of kernels, terrible outcomes for another, and no known reliable heuristic or modeling they could use to automatically choose.

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#117

Earlier quoted context omitted.

what kind of AI are you using that generates shitty commit messages? This a common kind of message from Claude / Augment: Fix dynamic channel list by passing auth via metadata - Pass userId and userEmail in metadata when calling HTTP transport - AuthenticatedToolsProviderFactory now reads from context.metadata - Each tools/list request creates a fresh ToolsProvider with authentication - Execute command description no…

> - Tested locally and working correctly This is completely meaningless and just pollutes the log.

And there's at least an 80% chance one of those items is, in fact, not in the commit.

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#118

Earlier quoted context omitted.

Or Intel checking for "GenuineIntel" in ICC's output: https://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Support...

Or Win 3.1 looking for whatever shibboleth was in MS-DOS and popping up a scary-looking message if it found another DOS? https://en.wikipedia.org/wiki/AARD_code

I don’t think anybody remembers this since that code never shipped in retail.

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#119
post #17

Heh. Does anyone remember when almost 25 years ago ATI (AMD) caught manipulating the Quake III benchmarks by renaming the executables to ‘quack’? https://web.archive.org/web/20230929180112/https://techrepor... https://web.archive.org/web/20011108190056/https://hardocp.c... https://web.archive.org/web/20011118183932/www.3dcenter.de/a...

Just in case anyone else parsed that sentence the same way as me, ati detected "quake" as the executable and changed things like texture quality etc to increase benchmark performance. Some people discovered this after they renamed the executable to "quack" and the image quality improved but the benchmarks were lower, proving that the ati drivers "optimised" by reducing quality. Ati did not rename quake to quack as I…

The story was that they used a lower mipmap level (blurrier textures) when the process was named Quake, but used the normal mipmap level (standard textures) when the process was named Quack.

Re: Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it

#120

Earlier quoted context omitted.

Every vendor does this to this day - and its a morally grey practice, drivers hijack and modify the rendering loops of popular games, fixing bugs, replacing shaders with more optimized versions, enabling faster codepaths in the driver etc. These changes are supposed to have minimal to no impact on the actual output, but sometimes vendors are really aggressive, and significantly degrade the outputs so that the game ca…

> but sometimes vendors are really aggressive, and significantly degrade the outputs so that the game can run faster on their hardware. Do you have a source for this? I’d like to see some examples

Nvidia has a control panel with it's drivers. Open it up -> Manage 3D settings -> Program Settings. Scroll through and see how every single program/game you have installed openly has different defaults in it based on application name. As someone noted above others do the same thing.

Eg. Frostpunk has Antialiasing for transparency layers on. Slay the spire does not. I never set these settings. Nvidia literally does a lookup on first run for what they judge as best defaults and sets these appropriately.

Every single game/program you install has different options from a huge list of possible optimizations.

Post reply on HN